2023-10-03T17:17:38.918 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T17:17:38.919 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rraja@teuthology --archive /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690 --name rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi --verbose --description rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} -- /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/orig.config.yaml 2023-10-03T17:17:38.949 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T17:17:39.036 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690 branch: wip-ajarr-63009-only-reacquire-kickstart description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} email: rraja@redhat.com first_in_suite: false job_id: '7409690' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-ajarr-63009-only-reacquire-kickstart ceph: conf: client: rbd default features: 1 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 debug rbd: 20 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log global: osd crush chooseleaf type: 0 osd pool default pg num: 128 osd pool default pgp num: 128 osd pool default size: 2 mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-ajarr-63009-only-reacquire-kickstart sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad owner: scheduled_rraja@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 8213 sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad sleep_before_teardown: 0 suite: rbd:cli suite_branch: wip-ajarr-63009-only-reacquire-kickstart suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: null - ceph: null - install: extra_system_packages: - fio - workunit: clients: client.0: - rbd/rbd_support_module_recovery.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-03_17:10:36 tube: smithi user: rraja verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-03T17:17:39.036 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa; will attempt to use it 2023-10-03T17:17:39.037 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks 2023-10-03T17:17:39.037 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-03T17:17:39.038 INFO:teuthology.task.internal:Checking packages... 2023-10-03T17:17:39.061 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad' 2023-10-03T17:17:39.062 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-03T17:17:39.062 INFO:teuthology.packaging:ref: None 2023-10-03T17:17:39.062 INFO:teuthology.packaging:tag: None 2023-10-03T17:17:39.062 INFO:teuthology.packaging:branch: wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T17:17:39.062 INFO:teuthology.packaging:sha1: a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:17:39.062 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-ajarr-63009-only-reacquire-kickstart 2023-10-03T17:17:39.279 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6336.ga7de91bf 2023-10-03T17:17:39.279 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-03T17:17:39.286 INFO:teuthology.task.internal:no buildpackages task found 2023-10-03T17:17:39.286 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-03T17:17:39.292 INFO:teuthology.task.internal:Saving configuration 2023-10-03T17:17:39.303 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-03T17:17:39.309 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T17:17:39.334 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 17:11:34.330721', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-03T17:17:39.335 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-03T17:17:39.341 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2023-10-03T17:17:39.341 INFO:teuthology.run_tasks:Running task console_log... 2023-10-03T17:17:39.381 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2eccbeeaf0>, signals=[15]) 2023-10-03T17:17:39.382 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-03T17:17:39.388 INFO:teuthology.task.internal:Opening connections... 2023-10-03T17:17:39.388 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T17:17:39.390 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:17:39.457 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-03T17:17:39.463 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-10-03T17:17:39.527 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-10-03T17:17:39.528 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-10-03T17:17:39.588 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-03T17:17:39.589 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-10-03T17:17:39.590 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-03T17:17:39.615 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-03T17:17:39.623 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-03T17:17:39.699 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-03T17:17:39.699 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-10-03T17:17:39.719 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-03T17:17:39.725 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-03T17:17:39.725 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-10-03T17:17:39.788 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-03T17:17:39.821 INFO:teuthology.run_tasks:Running task kernel... 2023-10-03T17:17:39.835 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-03T17:17:39.835 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-03T17:17:39.835 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T17:17:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:17:39.854 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-03T17:17:39.916 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-372.9.1.el8.x86_64 2023-10-03T17:17:39.916 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-372.9.1.el8.x86_64 2023-10-03T17:17:39.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-10-03T17:17:41.068 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:17:41.068 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-10-03T17:17:42.528 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-10-03T17:17:45.020 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 16 MB 00:02 2023-10-03T17:17:48.712 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 409 kB/s | 24 kB 00:00 2023-10-03T17:17:49.423 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-10-03T17:17:49.473 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:17:49.474 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:17:49.474 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:17:49.580 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2023-10-03T17:17:49.931 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:17:49.931 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-10-03T17:17:50.202 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Tue 03 Oct 2023 05:17:48 PM UTC. 2023-10-03T17:17:50.248 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-03T17:17:50.290 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-10-03T17:17:50.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2023-10-03T17:17:50.687 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:17:50.687 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-10-03T17:17:50.953 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Tue 03 Oct 2023 05:17:48 PM UTC. 2023-10-03T17:17:50.997 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-10-03T17:17:51.030 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-10-03T17:17:51.081 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T17:17:51.163 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-10-03T17:17:51.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-10-03T17:17:51.164 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-10-03T17:17:51.164 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-10-03T17:17:51.164 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T17:17:51.194 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-10-03T17:17:51.208 DEBUG:teuthology.parallel:result is None 2023-10-03T17:17:51.208 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-03T17:17:51.217 INFO:teuthology.task.internal:Creating test directory... 2023-10-03T17:17:51.217 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T17:17:51.266 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-03T17:17:51.272 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-03T17:17:51.279 INFO:teuthology.task.internal:Creating archive directory... 2023-10-03T17:17:51.279 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T17:17:51.326 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-03T17:17:51.333 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-03T17:17:51.334 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T17:17:51.414 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T17:17:51.425 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T17:17:51.426 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-03T17:17:51.433 INFO:teuthology.task.internal:Configuring sudo... 2023-10-03T17:17:51.434 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T17:17:51.461 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-03T17:17:51.470 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-03T17:17:51.471 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T17:17:51.519 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T17:17:51.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T17:17:51.653 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T17:17:51.738 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T17:17:51.779 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:17:51.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T17:17:51.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-10-03T17:17:51.975 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T17:17:52.413 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-03T17:17:52.422 INFO:teuthology.task.internal:Starting timer... 2023-10-03T17:17:52.422 INFO:teuthology.run_tasks:Running task pcp... 2023-10-03T17:17:52.446 INFO:teuthology.run_tasks:Running task selinux... 2023-10-03T17:17:52.454 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-03T17:17:52.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-10-03T17:17:52.509 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-10-03T17:17:52.511 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-03T17:17:52.511 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-10-03T17:17:52.531 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-10-03T17:17:52.531 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive'} 2023-10-03T17:17:52.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T17:17:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:17:52.599 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-03T17:17:52.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-10-03T17:17:52.666 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-03T17:17:52.675 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-03T17:17:52.675 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-03T17:17:52.690 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-03T17:17:52.691 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-03T17:27:37.107 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com')] 2023-10-03T17:27:37.108 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T17:27:37.109 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T17:27:37.181 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-03T17:27:37.249 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T17:27:37.249 INFO:teuthology.run_tasks:Running task clock... 2023-10-03T17:27:37.260 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-03T17:27:37.261 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T17:27:37.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T17:27:37.329 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-03T17:27:37.345 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-03T17:27:37.427 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-10-03T17:27:37.439 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-10-03T17:27:37.455 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-03T17:27:37.470 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-03T17:27:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T17:27:37.541 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T17:27:37.542 INFO:teuthology.run_tasks:Running task install... 2023-10-03T17:27:37.555 DEBUG:teuthology.task.install:project ceph 2023-10-03T17:27:37.555 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T17:27:37.555 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T17:27:37.556 INFO:teuthology.task.install:Using flavor: default 2023-10-03T17:27:37.564 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T17:27:37.571 INFO:teuthology.task.install:extra packages: [] 2023-10-03T17:27:37.580 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'tag': None, 'wait_for_package': False} 2023-10-03T17:27:37.581 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:27:37.733 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/ 2023-10-03T17:27:37.734 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6336.ga7de91bf 2023-10-03T17:27:37.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-03T17:27:37.920 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:27:37.920 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-03T17:27:37.955 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-10-03T17:27:37.956 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-03T17:27:38.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-03T17:27:38.115 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-03T17:27:38.183 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-10-03T17:27:38.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-10-03T17:27:38.577 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:27:38.841 INFO:teuthology.orchestra.run.smithi031.stdout:44 files removed 2023-10-03T17:27:38.888 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-10-03T17:27:39.239 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:27:40.387 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 103 kB/s | 81 kB 00:00 2023-10-03T17:27:41.115 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-10-03T17:27:41.879 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 2.0 kB/s | 1.5 kB 00:00 2023-10-03T17:27:42.030 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-03T17:27:43.505 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-03T17:27:47.215 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-10-03T17:27:48.192 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 53 MB 00:00 2023-10-03T17:27:57.176 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 53 MB/s | 47 MB 00:00 2023-10-03T17:28:08.905 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 14 M 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-10-03T17:28:08.908 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 3.7 M 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 3.7 M 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 5.4 M 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 23 M 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 24 k 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-03T17:28:08.909 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 832 k 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 519 k 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 6.9 M 2023-10-03T17:28:08.910 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-03T17:28:08.911 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-03T17:28:08.911 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 45 k 2023-10-03T17:28:08.911 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 125 k 2023-10-03T17:28:08.911 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 197 k 2023-10-03T17:28:08.911 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 370 k 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 366 k 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 107 k 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout:Install 21 Packages 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-10-03T17:28:08.912 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:08.913 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 62 M 2023-10-03T17:28:08.913 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:28:11.120 INFO:teuthology.orchestra.run.smithi031.stdout:(1/23): ceph-base-18.0.0-6336.ga7de91bf.el8.x86 3.0 MB/s | 5.4 MB 00:01 2023-10-03T17:28:11.351 INFO:teuthology.orchestra.run.smithi031.stdout:(2/23): ceph-common-18.0.0-6336.ga7de91bf.el8.x 11 MB/s | 23 MB 00:02 2023-10-03T17:28:11.376 INFO:teuthology.orchestra.run.smithi031.stdout:(3/23): ceph-selinux-18.0.0-6336.ga7de91bf.el8. 94 kB/s | 24 kB 00:00 2023-10-03T17:28:11.461 INFO:teuthology.orchestra.run.smithi031.stdout:(4/23): libcephfs2-18.0.0-6336.ga7de91bf.el8.x8 7.5 MB/s | 832 kB 00:00 2023-10-03T17:28:11.487 INFO:teuthology.orchestra.run.smithi031.stdout:(5/23): libradosstriper1-18.0.0-6336.ga7de91bf. 4.7 MB/s | 519 kB 00:00 2023-10-03T17:28:11.577 INFO:teuthology.orchestra.run.smithi031.stdout:(6/23): python3-ceph-argparse-18.0.0-6336.ga7de 487 kB/s | 45 kB 00:00 2023-10-03T17:28:11.661 INFO:teuthology.orchestra.run.smithi031.stdout:(7/23): python3-ceph-common-18.0.0-6336.ga7de91 1.5 MB/s | 125 kB 00:00 2023-10-03T17:28:11.745 INFO:teuthology.orchestra.run.smithi031.stdout:(8/23): python3-cephfs-18.0.0-6336.ga7de91bf.el 2.3 MB/s | 197 kB 00:00 2023-10-03T17:28:11.844 INFO:teuthology.orchestra.run.smithi031.stdout:(9/23): librgw2-18.0.0-6336.ga7de91bf.el8.x86_6 18 MB/s | 6.9 MB 00:00 2023-10-03T17:28:11.878 INFO:teuthology.orchestra.run.smithi031.stdout:(10/23): python3-rados-18.0.0-6336.ga7de91bf.el 2.7 MB/s | 370 kB 00:00 2023-10-03T17:28:12.053 INFO:teuthology.orchestra.run.smithi031.stdout:(11/23): python3-rbd-18.0.0-6336.ga7de91bf.el8. 1.7 MB/s | 366 kB 00:00 2023-10-03T17:28:12.070 INFO:teuthology.orchestra.run.smithi031.stdout:(12/23): python3-rgw-18.0.0-6336.ga7de91bf.el8. 558 kB/s | 107 kB 00:00 2023-10-03T17:28:12.404 INFO:teuthology.orchestra.run.smithi031.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 176 kB/s | 59 kB 00:00 2023-10-03T17:28:12.554 INFO:teuthology.orchestra.run.smithi031.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 612 kB/s | 306 kB 00:00 2023-10-03T17:28:12.571 INFO:teuthology.orchestra.run.smithi031.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 450 kB/s | 75 kB 00:00 2023-10-03T17:28:12.629 INFO:teuthology.orchestra.run.smithi031.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 802 kB/s | 47 kB 00:00 2023-10-03T17:28:12.671 INFO:teuthology.orchestra.run.smithi031.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-10-03T17:28:12.713 INFO:teuthology.orchestra.run.smithi031.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 1.0 MB/s | 44 kB 00:00 2023-10-03T17:28:12.755 INFO:teuthology.orchestra.run.smithi031.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-10-03T17:28:12.848 INFO:teuthology.orchestra.run.smithi031.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 5.9 MB/s | 1.7 MB 00:00 2023-10-03T17:28:13.289 INFO:teuthology.orchestra.run.smithi031.stdout:(21/23): librbd1-18.0.0-6336.ga7de91bf.el8.x86_ 8.4 MB/s | 3.7 MB 00:00 2023-10-03T17:28:13.377 INFO:teuthology.orchestra.run.smithi031.stdout:(22/23): librados2-18.0.0-6336.ga7de91bf.el8.x8 6.1 MB/s | 3.7 MB 00:00 2023-10-03T17:28:14.431 INFO:teuthology.orchestra.run.smithi031.stdout:(23/23): ceph-radosgw-18.0.0-6336.ga7de91bf.el8 2.7 MB/s | 14 MB 00:05 2023-10-03T17:28:14.434 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:28:14.434 INFO:teuthology.orchestra.run.smithi031.stdout:Total 11 MB/s | 62 MB 00:05 2023-10-03T17:28:14.483 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:28:14.533 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:28:14.534 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:28:14.897 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:28:14.901 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:28:16.135 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:28:16.341 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-10-03T17:28:16.542 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-03T17:28:16.963 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-10-03T17:28:17.159 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 3/25 2023-10-03T17:28:17.241 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 3/25 2023-10-03T17:28:17.440 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.0.0-6336.ga7de91bf.el8.x86_64 4/25 2023-10-03T17:28:17.841 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-10-03T17:28:18.019 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 6/25 2023-10-03T17:28:18.451 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 6/25 2023-10-03T17:28:18.609 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 7/25 2023-10-03T17:28:18.699 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 7/25 2023-10-03T17:28:18.842 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-03T17:28:18.918 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-10-03T17:28:19.090 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-10-03T17:28:19.421 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.0.0-6336.ga7de91bf.el8. 10/25 2023-10-03T17:28:19.623 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.0.0-6336.ga7de91bf.el8.x86_64 11/25 2023-10-03T17:28:19.862 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 12/25 2023-10-03T17:28:20.041 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.0.0-6336.ga7de91bf.el8.x86_6 13/25 2023-10-03T17:28:20.113 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6336.ga7de91bf.el8.x86_6 13/25 2023-10-03T17:28:20.285 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-10-03T17:28:20.484 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-10-03T17:28:21.336 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-10-03T17:28:21.508 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.0.0-6336.ga7de91bf.el8.x86_64 17/25 2023-10-03T17:28:21.585 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.0.0-6336.ga7de91bf.el8.x86_64 17/25 2023-10-03T17:28:21.838 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 18/25 2023-10-03T17:28:22.008 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.0.0-6336.ga7de91bf.el8.x8 19/25 2023-10-03T17:28:25.984 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 20/25 2023-10-03T17:28:26.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 20/25 2023-10-03T17:28:26.850 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 20/25 2023-10-03T17:28:27.151 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 21/25 2023-10-03T17:28:27.237 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 21/25 2023-10-03T17:28:27.398 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.0.0-6336.ga7de91bf.el8.x86_64 22/25 2023-10-03T17:28:46.577 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6336.ga7de91bf.el8.x86_64 22/25 2023-10-03T17:28:46.577 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-10-03T17:28:46.577 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-10-03T17:28:46.577 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-10-03T17:28:46.577 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /run 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-10-03T17:28:46.578 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:48.139 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 23/25 2023-10-03T17:28:48.162 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 23/25 2023-10-03T17:28:48.162 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-03T17:28:48.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-10-03T17:28:48.389 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-03T17:28:49.036 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 2/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 3/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.0.0-6336.ga7de91bf.el8.x86_64 4/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 5/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.0.0-6336.ga7de91bf.el8.x86_6 6/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.0.0-6336.ga7de91bf.el8.x86_64 7/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6336.ga7de91bf.el8. 8/25 2023-10-03T17:28:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.0.0-6336.ga7de91bf.el8.x8 9/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.0.0-6336.ga7de91bf.el8.x86_64 10/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.0.0-6336.ga7de91bf.el8.x86_64 11/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 12/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 13/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-10-03T17:28:49.038 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-10-03T17:28:49.039 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 22/25 2023-10-03T17:28:49.039 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-10-03T17:28:49.039 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 24/25 2023-10-03T17:28:50.468 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-10-03T17:28:50.468 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:54 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:28:53.001 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:28:53.222 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:53.223 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-10-03T17:28:53.223 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.223 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.223 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:28:53.223 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.224 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.224 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.224 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-03T17:28:53.225 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:53.226 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:28:53.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-10-03T17:28:53.764 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:28:54.592 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:58 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 49 M 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-10-03T17:28:56.149 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 184 k 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 50 M 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 222 M 2023-10-03T17:28:56.150 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:28:56.197 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-10-03T17:28:56.273 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.8 MB/s | 323 kB 00:00 2023-10-03T17:28:56.314 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2023-10-03T17:28:56.357 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-10-03T17:28:56.491 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): libcephsqlite-18.0.0-6336.ga7de91bf.el8. 539 kB/s | 184 kB 00:00 2023-10-03T17:28:59.240 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-test-18.0.0-6336.ga7de91bf.el8.x86_ 16 MB/s | 49 MB 00:03 2023-10-03T17:28:59.242 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:28:59.242 INFO:teuthology.orchestra.run.smithi031.stdout:Total 16 MB/s | 50 MB 00:03 2023-10-03T17:28:59.281 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:28:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:28:59.305 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:28:59.590 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:28:59.594 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:29:00.062 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:29:00.201 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-03T17:29:00.278 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-10-03T17:29:00.516 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-10-03T17:29:00.678 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-10-03T17:29:00.834 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-10-03T17:29:01.039 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.0.0-6336.ga7de91bf.el8.x86_64 5/6 2023-10-03T17:29:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6336.ga7de91bf.el8.x86_64 5/6 2023-10-03T17:29:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.0.0-6336.ga7de91bf.el8.x86_64 6/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.0.0-6336.ga7de91bf.el8.x86_64 6/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.0.0-6336.ga7de91bf.el8.x86_64 2/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-10-03T17:29:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-10-03T17:29:07.643 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-10-03T17:29:07.644 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:11 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:10.126 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:29:10.328 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:10.328 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:29:10.328 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:10.328 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-3.el8.x86_64 2023-10-03T17:29:10.328 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:10.329 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-03T17:29:10.329 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-1.el8.x86_64 2023-10-03T17:29:10.329 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-03T17:29:10.329 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:10.329 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:10.493 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-10-03T17:29:10.850 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:11.738 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:15 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:13.301 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:29:13.304 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-10-03T17:29:13.305 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:29:13.305 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-10-03T17:29:13.305 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:29:13.305 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 6.4 k 2023-10-03T17:29:13.305 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 2.3 M 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 1.6 M 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 242 k 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 4.6 M 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 18 M 2023-10-03T17:29:13.306 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-03T17:29:13.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-03T17:29:13.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-10-03T17:29:13.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-03T17:29:13.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-03T17:29:13.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-03T17:29:13.308 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-03T17:29:13.309 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-03T17:29:13.310 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-03T17:29:13.310 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-03T17:29:13.310 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-03T17:29:13.310 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-03T17:29:13.310 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:29:13.311 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-10-03T17:29:13.312 INFO:teuthology.orchestra.run.smithi031.stdout:Install 32 Packages 2023-10-03T17:29:13.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 30 M 2023-10-03T17:29:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 121 M 2023-10-03T17:29:13.313 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:29:13.491 INFO:teuthology.orchestra.run.smithi031.stdout:(1/32): ceph-18.0.0-6336.ga7de91bf.el8.x86_64.r 53 kB/s | 6.4 kB 00:00 2023-10-03T17:29:13.641 INFO:teuthology.orchestra.run.smithi031.stdout:(2/32): ceph-mgr-18.0.0-6336.ga7de91bf.el8.x86_ 6.0 MB/s | 1.6 MB 00:00 2023-10-03T17:29:14.235 INFO:teuthology.orchestra.run.smithi031.stdout:(3/32): ceph-mds-18.0.0-6336.ga7de91bf.el8.x86_ 2.7 MB/s | 2.3 MB 00:00 2023-10-03T17:29:14.300 INFO:teuthology.orchestra.run.smithi031.stdout:(4/32): ceph-mon-18.0.0-6336.ga7de91bf.el8.x86_ 5.7 MB/s | 4.6 MB 00:00 2023-10-03T17:29:14.326 INFO:teuthology.orchestra.run.smithi031.stdout:(5/32): ceph-mgr-modules-core-18.0.0-6336.ga7de 2.6 MB/s | 242 kB 00:00 2023-10-03T17:29:14.718 INFO:teuthology.orchestra.run.smithi031.stdout:(6/32): ceph-osd-18.0.0-6336.ga7de91bf.el8.x86_ 17 MB/s | 18 MB 00:01 2023-10-03T17:29:14.743 INFO:teuthology.orchestra.run.smithi031.stdout:(7/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 99 kB/s | 44 kB 00:00 2023-10-03T17:29:14.909 INFO:teuthology.orchestra.run.smithi031.stdout:(8/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 317 kB/s | 185 kB 00:00 2023-10-03T17:29:15.059 INFO:teuthology.orchestra.run.smithi031.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 75 kB/s | 11 kB 00:00 2023-10-03T17:29:15.135 INFO:teuthology.orchestra.run.smithi031.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 980 kB/s | 384 kB 00:00 2023-10-03T17:29:15.310 INFO:teuthology.orchestra.run.smithi031.stdout:(11/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 291 kB/s | 173 kB 00:00 2023-10-03T17:29:15.594 INFO:teuthology.orchestra.run.smithi031.stdout:(12/32): python3-jaraco-functools-2.0-4.el8.noa 33 kB/s | 18 kB 00:00 2023-10-03T17:29:15.619 INFO:teuthology.orchestra.run.smithi031.stdout:(13/32): python3-logutils-0.3.5-11.el8.noarch.r 101 kB/s | 49 kB 00:00 2023-10-03T17:29:15.644 INFO:teuthology.orchestra.run.smithi031.stdout:(14/32): python3-more-itertools-7.2.0-3.el8.noa 176 kB/s | 59 kB 00:00 2023-10-03T17:29:15.761 INFO:teuthology.orchestra.run.smithi031.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 110 kB/s | 16 kB 00:00 2023-10-03T17:29:15.786 INFO:teuthology.orchestra.run.smithi031.stdout:(16/32): python3-simplegeneric-0.8.1-17.el8.noa 132 kB/s | 19 kB 00:00 2023-10-03T17:29:15.823 INFO:teuthology.orchestra.run.smithi031.stdout:(17/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-10-03T17:29:15.887 INFO:teuthology.orchestra.run.smithi031.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2023-10-03T17:29:15.912 INFO:teuthology.orchestra.run.smithi031.stdout:(19/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-10-03T17:29:15.937 INFO:teuthology.orchestra.run.smithi031.stdout:(20/32): python3-trustme-0.6.0-4.el8.noarch.rpm 235 kB/s | 27 kB 00:00 2023-10-03T17:29:16.039 INFO:teuthology.orchestra.run.smithi031.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2023-10-03T17:29:16.055 INFO:teuthology.orchestra.run.smithi031.stdout:(22/32): python3-webtest-2.0.33-1.el8.noarch.rp 729 kB/s | 85 kB 00:00 2023-10-03T17:29:16.106 INFO:teuthology.orchestra.run.smithi031.stdout:(23/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.3 MB/s | 251 kB 00:00 2023-10-03T17:29:16.130 INFO:teuthology.orchestra.run.smithi031.stdout:(24/32): python3-pycparser-2.14-14.el8.noarch.r 1.4 MB/s | 109 kB 00:00 2023-10-03T17:29:16.156 INFO:teuthology.orchestra.run.smithi031.stdout:(25/32): python3-zc-lockfile-2.0-2.el8.noarch.r 196 kB/s | 23 kB 00:00 2023-10-03T17:29:16.231 INFO:teuthology.orchestra.run.smithi031.stdout:(26/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-10-03T17:29:16.248 INFO:teuthology.orchestra.run.smithi031.stdout:(27/32): python3-pytz-2017.2-9.el8.noarch.rpm 591 kB/s | 54 kB 00:00 2023-10-03T17:29:16.265 INFO:teuthology.orchestra.run.smithi031.stdout:(28/32): python3-markupsafe-0.23-19.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-10-03T17:29:16.294 INFO:teuthology.orchestra.run.smithi031.stdout:(29/32): python3-mako-1.0.6-13.el8.noarch.rpm 3.7 MB/s | 157 kB 00:00 2023-10-03T17:29:16.318 INFO:teuthology.orchestra.run.smithi031.stdout:(30/32): python3-cryptography-3.2.1-5.el8.x86_6 2.6 MB/s | 559 kB 00:00 2023-10-03T17:29:16.340 INFO:teuthology.orchestra.run.smithi031.stdout:(31/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 2.0 MB/s | 103 kB 00:00 2023-10-03T17:29:16.365 INFO:teuthology.orchestra.run.smithi031.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 4.4 MB/s | 457 kB 00:00 2023-10-03T17:29:16.370 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:29:16.370 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.9 MB/s | 30 MB 00:03 2023-10-03T17:29:16.466 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:29:16.507 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:29:16.507 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:29:16.732 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:29:16.736 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:29:17.831 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:29:18.074 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-10-03T17:29:18.363 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-10-03T17:29:18.561 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-10-03T17:29:18.797 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-10-03T17:29:19.015 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-10-03T17:29:19.187 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-10-03T17:29:19.386 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-10-03T17:29:19.784 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-10-03T17:29:20.061 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-10-03T17:29:20.223 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-10-03T17:29:20.427 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-10-03T17:29:20.658 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-10-03T17:29:20.903 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-10-03T17:29:21.073 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-10-03T17:29:21.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-10-03T17:29:21.411 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-10-03T17:29:21.568 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-10-03T17:29:22.238 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-10-03T17:29:22.456 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-10-03T17:29:22.681 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-10-03T17:29:22.871 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-10-03T17:29:23.160 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-10-03T17:29:23.415 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-10-03T17:29:23.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-10-03T17:29:24.032 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-10-03T17:29:24.345 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-10-03T17:29:24.677 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6336.ga7de91bf.el8. 27/32 2023-10-03T17:29:24.824 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 28/32 2023-10-03T17:29:26.540 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 28/32 2023-10-03T17:29:26.724 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.0.0-6336.ga7de91bf.el8.x86_64 29/32 2023-10-03T17:29:27.212 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.0.0-6336.ga7de91bf.el8.x86_64 29/32 2023-10-03T17:29:27.405 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.0.0-6336.ga7de91bf.el8.x86_64 30/32 2023-10-03T17:29:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.0.0-6336.ga7de91bf.el8.x86_64 30/32 2023-10-03T17:29:27.789 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.0.0-6336.ga7de91bf.el8.x86_64 31/32 2023-10-03T17:29:27.807 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.0.0-6336.ga7de91bf.el8.x86_64 31/32 2023-10-03T17:29:27.946 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.0.0-6336.ga7de91bf.el8.x86_64 32/32 2023-10-03T17:29:28.328 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.0.0-6336.ga7de91bf.el8.x86_64 32/32 2023-10-03T17:29:28.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/32 2023-10-03T17:29:28.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.0.0-6336.ga7de91bf.el8.x86_64 2/32 2023-10-03T17:29:28.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 3/32 2023-10-03T17:29:28.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.0.0-6336.ga7de91bf.el8.x86_64 4/32 2023-10-03T17:29:28.329 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.0.0-6336.ga7de91bf.el8.x86_64 5/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6336.ga7de91bf.el8. 6/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-10-03T17:29:28.330 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-10-03T17:29:28.332 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-10-03T17:29:28.332 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-10-03T17:29:28.332 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-10-03T17:29:28.333 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-10-03T17:29:28.334 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-10-03T17:29:29.874 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-10-03T17:29:29.874 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:33 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:32.802 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:33.005 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-03T17:29:33.006 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-2.6-1.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.14-14.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-03T17:29:33.007 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-03T17:29:33.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-03T17:29:33.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-03T17:29:33.008 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-03T17:29:33.008 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:33.008 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:33.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-10-03T17:29:33.595 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:34.468 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:38 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:35.911 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:29:36.027 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:29:36.029 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:29:36.029 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:36.130 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-10-03T17:29:36.497 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:37.932 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:41 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:39.505 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 181 k 2023-10-03T17:29:39.506 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 181 k 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 729 k 2023-10-03T17:29:39.507 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:29:40.060 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.0.0-6336.ga7de91bf.el8.noarch.rpm 327 kB/s | 181 kB 00:00 2023-10-03T17:29:40.061 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:29:40.061 INFO:teuthology.orchestra.run.smithi031.stdout:Total 326 kB/s | 181 kB 00:00 2023-10-03T17:29:40.062 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:29:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:29:40.071 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:29:40.073 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:29:40.077 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:29:40.120 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:29:41.907 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:29:42.042 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:29:42.218 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:29:43.285 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:29:43.285 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:47 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:45.864 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:29:46.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:46.062 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:29:46.062 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:29:46.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:46.062 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:46.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-10-03T17:29:46.588 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:47.465 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:51 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:49.035 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:29:49.036 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:49.036 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-03T17:29:49.036 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 167 k 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:29:49.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:49.038 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 167 k 2023-10-03T17:29:49.038 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 512 k 2023-10-03T17:29:49.038 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:29:49.555 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.0.0-6336.ga7de91 322 kB/s | 167 kB 00:00 2023-10-03T17:29:49.556 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:29:49.556 INFO:teuthology.orchestra.run.smithi031.stdout:Total 322 kB/s | 167 kB 00:00 2023-10-03T17:29:49.556 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:29:49.570 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:29:49.570 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:29:49.627 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:29:49.631 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:29:49.918 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:29:50.164 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6336.ga7de91bf. 1/1 2023-10-03T17:29:50.533 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6336.ga7de91bf. 1/1 2023-10-03T17:29:51.581 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6336.ga7de91bf. 1/1 2023-10-03T17:29:51.582 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:55 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:54.307 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:29:54.502 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:54.503 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:29:54.503 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:29:54.503 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:29:54.503 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:54.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-10-03T17:29:55.005 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:55.995 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:59 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:29:57.426 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:29:57.542 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:29:57.543 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:29:57.544 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:29:57.657 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-10-03T17:29:58.021 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:29:58.907 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:02 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:00.408 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 1.7 M 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 23 k 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 15 k 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-03T17:30:00.409 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-10-03T17:30:00.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:00.411 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.1 M 2023-10-03T17:30:00.411 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.5 M 2023-10-03T17:30:00.411 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:30:00.702 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): ceph-grafana-dashboards-18.0.0-6336.ga7d 97 kB/s | 23 kB 00:00 2023-10-03T17:30:00.727 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): ceph-prometheus-alerts-18.0.0-6336.ga7de 55 kB/s | 15 kB 00:00 2023-10-03T17:30:00.861 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): ceph-mgr-dashboard-18.0.0-6336.ga7de91bf 4.4 MB/s | 1.7 MB 00:00 2023-10-03T17:30:00.886 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2023-10-03T17:30:00.912 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.0 MB/s | 196 kB 00:00 2023-10-03T17:30:01.078 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 196 kB/s | 43 kB 00:00 2023-10-03T17:30:01.081 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:30:01.081 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.1 MB/s | 2.1 MB 00:00 2023-10-03T17:30:01.094 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:30:01.104 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:30:01.105 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:30:01.166 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:30:01.173 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:30:01.448 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:30:01.618 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-10-03T17:30:01.783 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-10-03T17:30:01.935 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-10-03T17:30:02.116 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6336.ga7de91bf.el8.n 4/6 2023-10-03T17:30:02.464 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6336.ga7de91bf.el8. 5/6 2023-10-03T17:30:02.691 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6336.ga7de91bf.el8.noarc 6/6 2023-10-03T17:30:02.926 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6336.ga7de91bf.el8.noarc 6/6 2023-10-03T17:30:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6336.ga7de91bf.el8. 1/6 2023-10-03T17:30:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6336.ga7de91bf.el8.noarc 2/6 2023-10-03T17:30:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6336.ga7de91bf.el8.n 3/6 2023-10-03T17:30:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-10-03T17:30:02.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-10-03T17:30:03.999 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-10-03T17:30:03.999 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:07 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:09.300 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-03T17:30:09.505 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-03T17:30:09.506 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:09.506 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:30:09.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-10-03T17:30:10.058 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:30:11.605 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:15 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:13.185 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:30:13.186 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 7.4 M 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-03T17:30:13.187 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout:Install 8 Packages 2023-10-03T17:30:13.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:13.189 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 35 M 2023-10-03T17:30:13.189 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 210 M 2023-10-03T17:30:13.189 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:30:13.294 INFO:teuthology.orchestra.run.smithi031.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.6 MB/s | 171 kB 00:00 2023-10-03T17:30:13.400 INFO:teuthology.orchestra.run.smithi031.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 3.0 MB/s | 644 kB 00:00 2023-10-03T17:30:13.551 INFO:teuthology.orchestra.run.smithi031.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 25 MB/s | 3.7 MB 00:00 2023-10-03T17:30:13.659 INFO:teuthology.orchestra.run.smithi031.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2023-10-03T17:30:13.827 INFO:teuthology.orchestra.run.smithi031.stdout:(5/8): python3-scipy-1.0.0-21.module+el8.5.0+10 26 MB/s | 14 MB 00:00 2023-10-03T17:30:13.910 INFO:teuthology.orchestra.run.smithi031.stdout:(6/8): ceph-mgr-diskprediction-local-18.0.0-633 10 MB/s | 7.4 MB 00:00 2023-10-03T17:30:14.035 INFO:teuthology.orchestra.run.smithi031.stdout:(7/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.9 MB 00:00 2023-10-03T17:30:14.102 INFO:teuthology.orchestra.run.smithi031.stdout:(8/8): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-10-03T17:30:14.102 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:30:14.103 INFO:teuthology.orchestra.run.smithi031.stdout:Total 39 MB/s | 35 MB 00:00 2023-10-03T17:30:14.350 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:30:14.368 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:30:14.368 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:30:14.678 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:30:14.682 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:30:15.246 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:30:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-03T17:30:15.510 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-10-03T17:30:15.664 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-03T17:30:16.317 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-03T17:30:16.472 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-03T17:30:17.105 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-10-03T17:30:17.305 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-10-03T17:30:18.035 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-10-03T17:30:18.329 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-03T17:30:18.522 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-03T17:30:20.395 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-03T17:30:21.684 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-10-03T17:30:21.938 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6336.ga7de91b 8/8 2023-10-03T17:30:22.199 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6336.ga7de91b 8/8 2023-10-03T17:30:22.199 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6336.ga7de91b 1/8 2023-10-03T17:30:22.199 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-10-03T17:30:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-10-03T17:30:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-10-03T17:30:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-10-03T17:30:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-10-03T17:30:22.200 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-10-03T17:30:23.287 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-10-03T17:30:23.287 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:27 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:28.674 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.15-3.el8.x86_64 2023-10-03T17:30:28.876 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-10-03T17:30:28.877 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-10-03T17:30:28.877 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-10-03T17:30:28.877 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-10-03T17:30:28.877 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:28.877 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:30:29.022 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-10-03T17:30:29.394 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:30:30.932 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:34 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:32.510 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 47 k 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:30:32.511 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-03T17:30:32.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:32.513 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 1.9 M 2023-10-03T17:30:32.514 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 25 M 2023-10-03T17:30:32.514 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:30:32.705 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-cachetools-3.1.1-4.el8.noarch.r 251 kB/s | 33 kB 00:00 2023-10-03T17:30:32.730 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-certifi-2018.10.15-7.el8.noarch 104 kB/s | 16 kB 00:00 2023-10-03T17:30:32.789 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): python3-google-auth-1.1.1-10.el8.noarch 974 kB/s | 81 kB 00:00 2023-10-03T17:30:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-rsa-4.9-2.el8.noarch.rpm 1.5 MB/s | 64 kB 00:00 2023-10-03T17:30:32.856 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): ceph-mgr-rook-18.0.0-6336.ga7de91bf.el8 168 kB/s | 47 kB 00:00 2023-10-03T17:30:32.890 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-kubernetes-11.0.0-6.el8.noarch. 7.3 MB/s | 1.2 MB 00:00 2023-10-03T17:30:32.949 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-websocket-client-0.56.0-5.el8.n 522 kB/s | 61 kB 00:00 2023-10-03T17:30:32.990 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 319 kB/s | 43 kB 00:00 2023-10-03T17:30:33.057 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 975 kB/s | 155 kB 00:00 2023-10-03T17:30:33.124 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-jsonpatch-1.21-2.el8.noarch.rp 211 kB/s | 27 kB 00:00 2023-10-03T17:30:33.158 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-pyasn1-modules-0.3.7-6.el8.noa 549 kB/s | 110 kB 00:00 2023-10-03T17:30:33.258 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 630 kB/s | 126 kB 00:00 2023-10-03T17:30:33.285 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 126 kB/s | 20 kB 00:00 2023-10-03T17:30:33.285 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:30:33.285 INFO:teuthology.orchestra.run.smithi031.stdout:Total 2.5 MB/s | 1.9 MB 00:00 2023-10-03T17:30:33.347 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:30:33.364 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:30:33.365 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:30:33.443 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:30:33.445 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:30:33.948 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:30:34.129 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-10-03T17:30:34.366 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-10-03T17:30:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-10-03T17:30:34.701 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-10-03T17:30:34.920 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-10-03T17:30:35.125 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-10-03T17:30:35.323 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-10-03T17:30:35.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-10-03T17:30:35.709 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-10-03T17:30:35.910 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-10-03T17:30:37.028 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-10-03T17:30:37.270 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-10-03T17:30:37.435 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.0.0-6336.ga7de91bf.el8.noarch 13/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6336.ga7de91bf.el8.noarch 13/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6336.ga7de91bf.el8.noarch 1/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-10-03T17:30:37.674 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-10-03T17:30:37.675 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-10-03T17:30:37.675 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-10-03T17:30:37.675 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-10-03T17:30:38.868 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-10-03T17:30:38.868 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:42 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:44.966 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-03T17:30:45.168 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:45.169 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:30:45.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-10-03T17:30:45.679 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:30:47.260 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:51 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:48.815 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:30:48.816 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-10-03T17:30:48.816 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:30:48.816 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-10-03T17:30:48.816 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 132 k 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2023-10-03T17:30:48.817 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:48.818 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.0 M 2023-10-03T17:30:48.818 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-10-03T17:30:48.818 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:30:49.106 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 521 kB/s | 60 kB 00:00 2023-10-03T17:30:49.164 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): ceph-mgr-cephadm-18.0.0-6336.ga7de91bf.e 756 kB/s | 132 kB 00:00 2023-10-03T17:30:49.198 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.2 MB/s | 476 kB 00:00 2023-10-03T17:30:49.290 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.9 MB/s | 538 kB 00:00 2023-10-03T17:30:49.392 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-10-03T17:30:49.392 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:30:49.392 INFO:teuthology.orchestra.run.smithi031.stdout:Total 10 MB/s | 6.0 MB 00:00 2023-10-03T17:30:49.445 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:30:49.457 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:30:49.457 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:30:49.540 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:30:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:30:50.601 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:30:50.951 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-10-03T17:30:51.161 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-10-03T17:30:51.515 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-03T17:30:51.767 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-10-03T17:30:51.991 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 5/5 2023-10-03T17:30:52.238 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 5/5 2023-10-03T17:30:52.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 1/5 2023-10-03T17:30:52.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-10-03T17:30:52.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-10-03T17:30:52.238 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-10-03T17:30:53.322 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-10-03T17:30:53.322 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:57 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:30:59.777 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:30:59.975 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.5.1-7.el8.noarch 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:30:59.976 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:00.089 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-10-03T17:31:00.465 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:01.871 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:05 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:03.408 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 912 k 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:03.409 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:31:03.410 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:03.410 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-10-03T17:31:03.410 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:03.410 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 995 k 2023-10-03T17:31:03.411 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.1 M 2023-10-03T17:31:03.411 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:31:03.577 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 495 kB/s | 83 kB 00:00 2023-10-03T17:31:03.685 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.0.0-6336.ga7de91bf.el8.x86_ 3.2 MB/s | 912 kB 00:00 2023-10-03T17:31:03.686 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:31:03.686 INFO:teuthology.orchestra.run.smithi031.stdout:Total 3.5 MB/s | 995 kB 00:00 2023-10-03T17:31:03.698 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:31:03.721 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:31:03.722 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:31:03.802 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:31:03.806 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:31:04.007 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:31:04.316 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-10-03T17:31:04.480 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 2/2 2023-10-03T17:31:04.816 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 2/2 2023-10-03T17:31:04.816 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/2 2023-10-03T17:31:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-10-03T17:31:05.899 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:09 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:10.859 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:31:11.058 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:11.062 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:31:11.063 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-10-03T17:31:11.063 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:11.063 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:11.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-10-03T17:31:11.573 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:13.223 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:17 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:14.742 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:14.743 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:14.743 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:31:14.743 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:14.744 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:31:14.744 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.0.0-6336.ga7de91bf.el8 ceph-noarch 261 k 2023-10-03T17:31:14.744 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 261 k 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.3 M 2023-10-03T17:31:14.749 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:31:15.084 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-18.0.0-6336.ga7de91bf.el8.noarch.rp 770 kB/s | 261 kB 00:00 2023-10-03T17:31:15.087 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:31:15.087 INFO:teuthology.orchestra.run.smithi031.stdout:Total 768 kB/s | 261 kB 00:00 2023-10-03T17:31:15.088 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:31:15.096 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:31:15.097 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:31:15.115 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:31:15.119 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:31:15.298 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:31:15.580 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:31:15.942 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:31:17.010 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.0.0-6336.ga7de91bf.el8.noarch 1/1 2023-10-03T17:31:17.010 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:21 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:19.953 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:31:20.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:20.152 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:31:20.152 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.0.0-6336.ga7de91bf.el8.noarch 2023-10-03T17:31:20.152 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:20.152 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:20.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-10-03T17:31:20.662 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:22.256 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:26 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:23.818 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 139 k 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:23.819 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:31:23.820 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:23.820 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 139 k 2023-10-03T17:31:23.820 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 513 k 2023-10-03T17:31:23.820 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:31:24.005 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.0.0-6336.ga7de91bf.el8.x86_64 754 kB/s | 139 kB 00:00 2023-10-03T17:31:24.005 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:31:24.005 INFO:teuthology.orchestra.run.smithi031.stdout:Total 749 kB/s | 139 kB 00:00 2023-10-03T17:31:24.006 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:31:24.012 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:31:24.012 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:31:24.070 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:31:24.074 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:31:24.314 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:31:24.492 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:24.734 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:25.830 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:25.830 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:29 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:29.086 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:31:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:29.283 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:31:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:31:29.283 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:29.283 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:29.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-10-03T17:31:29.794 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:30.734 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:34 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:32.138 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:32.255 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:32.256 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:32.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-10-03T17:31:32.754 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:33.646 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:37 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:35.202 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 31 k 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:35.203 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2023-10-03T17:31:35.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 139 k 2023-10-03T17:31:35.204 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:31:35.434 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.0.0-6336.ga7de91bf.el8.x86_6 134 kB/s | 31 kB 00:00 2023-10-03T17:31:35.435 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:31:35.435 INFO:teuthology.orchestra.run.smithi031.stdout:Total 134 kB/s | 31 kB 00:00 2023-10-03T17:31:35.435 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:31:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:31:35.440 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:31:35.452 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:31:35.455 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:31:35.528 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:31:35.698 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:35.924 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:36.963 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:31:36.963 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:40 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:39.780 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:31:39.979 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:39.980 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:31:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:31:39.980 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:31:39.980 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:40.111 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-10-03T17:31:40.492 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:42.051 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:46 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:43.477 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:43.597 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:43.599 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:43.599 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:43.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-10-03T17:31:44.078 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:45.686 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:49 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:47.117 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:47.233 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:47.234 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:47.234 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:47.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-10-03T17:31:47.727 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:49.104 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:53 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:50.520 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:50.636 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:50.638 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:50.638 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:50.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-10-03T17:31:51.120 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:52.725 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:56 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:54.153 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:54.270 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:54.272 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:54.272 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:54.388 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-10-03T17:31:54.770 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:56.231 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:00 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:31:57.666 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:31:57.787 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:31:57.787 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:31:57.787 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:31:57.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-10-03T17:31:58.277 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:31:59.826 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:03 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:01.250 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:32:01.371 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:32:01.373 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-10-03T17:32:01.373 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:32:01.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-10-03T17:32:02.005 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:32:03.589 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:07 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:05.130 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:32:05.130 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 94 k 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:05.131 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 94 k 2023-10-03T17:32:05.132 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 276 k 2023-10-03T17:32:05.132 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:32:05.288 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.0.0-6336.ga7de91bf.el8.x86_64.rpm 589 kB/s | 94 kB 00:00 2023-10-03T17:32:05.289 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:32:05.289 INFO:teuthology.orchestra.run.smithi031.stdout:Total 585 kB/s | 94 kB 00:00 2023-10-03T17:32:05.289 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:32:05.305 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:32:05.306 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:32:05.361 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:32:05.363 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:32:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:32:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:07.132 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:07.132 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:11 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:11.611 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:32:11.804 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:11.805 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:32:11.805 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:32:11.805 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:11.805 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:32:11.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-10-03T17:32:12.317 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:32:13.950 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:17 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:15.516 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 3.6 M 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:15.517 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.6 M 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 17 M 2023-10-03T17:32:15.518 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:32:15.910 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.0.0-6336.ga7de91bf.el8.x86_64.rpm 9.2 MB/s | 3.6 MB 00:00 2023-10-03T17:32:15.910 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:32:15.910 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.2 MB/s | 3.6 MB 00:00 2023-10-03T17:32:15.910 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:32:15.923 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:32:15.924 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:32:15.996 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:32:16.000 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:32:16.476 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:32:16.661 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:17.036 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:18.141 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:18.141 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:22 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:21.967 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:32:22.165 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:22.165 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:32:22.165 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:32:22.165 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:22.166 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:32:22.297 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-10-03T17:32:22.677 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:32:23.511 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:27 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:25.066 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.0.0-6336.ga7de91bf.el8 ceph 179 k 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:25.067 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 179 k 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 564 k 2023-10-03T17:32:25.068 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:32:25.254 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.0.0-6336.ga7de91bf.el8.x86_64.rpm 967 kB/s | 179 kB 00:00 2023-10-03T17:32:25.254 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:32:25.254 INFO:teuthology.orchestra.run.smithi031.stdout:Total 960 kB/s | 179 kB 00:00 2023-10-03T17:32:25.255 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:32:25.268 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:32:25.269 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:32:25.325 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:32:25.335 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:32:25.523 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:32:25.698 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:25.923 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:26.987 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 1/1 2023-10-03T17:32:26.987 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:30 ago on Tue 03 Oct 2023 05:27:56 PM UTC. 2023-10-03T17:32:29.602 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:32:29.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:29.802 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:32:29.802 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 2023-10-03T17:32:29.802 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:29.802 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:32:29.938 DEBUG:teuthology.parallel:result is None 2023-10-03T17:32:29.938 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:32:30.087 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-03T17:32:30.113 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-6336.ga7de91bf.el8 2023-10-03T17:32:30.114 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336.ga7de91bf.el8 2023-10-03T17:32:30.114 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336.ga7de91bf is installed. 2023-10-03T17:32:30.114 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-03T17:32:30.115 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:30.115 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-03T17:32:30.182 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-03T17:32:30.182 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:30.182 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-10-03T17:32:30.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-03T17:32:30.315 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-03T17:32:30.316 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:30.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-03T17:32:30.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-03T17:32:30.453 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-03T17:32:30.454 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:30.454 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-10-03T17:32:30.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-03T17:32:30.589 INFO:teuthology.run_tasks:Running task ceph... 2023-10-03T17:32:30.693 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-03T17:32:30.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-10-03T17:32:30.720 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-03T17:32:30.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-03T17:32:30.788 INFO:tasks.ceph:Creating extra log directories... 2023-10-03T17:32:30.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-03T17:32:30.863 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-03T17:32:30.864 INFO:tasks.ceph:config {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-03T17:32:30.864 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690', 'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'description': 'rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery}', 'email': 'rraja@redhat.com', 'first_in_suite': False, 'job_id': '7409690', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart'}, 'ceph': {'conf': {'client': {'rbd default features': 1}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'debug rbd': 20}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'global': {'osd crush chooseleaf type': 0, 'osd pool default pg num': 128, 'osd pool default pgp num': 128, 'osd pool default size': 2}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}}, 'owner': 'scheduled_rraja@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0']], 'seed': 8213, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'sleep_before_teardown': 0, 'suite': 'rbd:cli', 'suite_branch': 'wip-ajarr-63009-only-reacquire-kickstart', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'targets': {'smithi031.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'install': {'extra_system_packages': ['fio']}}, {'workunit': {'clients': {'client.0': ['rbd/rbd_support_module_recovery.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-03_17:10:36', 'tube': 'smithi', 'user': 'rraja', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-03T17:32:30.864 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-03T17:32:30.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-10-03T17:32:30.981 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:30.982 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T17:32:31.039 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T17:32:31.040 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-10-03T17:32:31.097 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T17:32:31.097 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:32:31.097 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 383831 Links: 1 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 17:32:16.831550584 +0000 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 17:26:40.463481396 +0000 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 17:26:40.463481396 +0000 2023-10-03T17:32:31.098 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T17:32:31.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T17:32:31.164 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T17:32:31.164 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T17:32:31.164 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000221141 s, 2.3 MB/s 2023-10-03T17:32:31.165 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T17:32:31.223 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-10-03T17:32:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T17:32:31.280 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:32:31.280 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 385646 Links: 1 2023-10-03T17:32:31.280 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:32:31.281 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T17:32:31.281 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 17:32:16.831550584 +0000 2023-10-03T17:32:31.281 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 17:26:41.583463277 +0000 2023-10-03T17:32:31.281 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 17:26:41.583463277 +0000 2023-10-03T17:32:31.281 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T17:32:31.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T17:32:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T17:32:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T17:32:31.347 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000221885 s, 2.3 MB/s 2023-10-03T17:32:31.348 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T17:32:31.406 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 378648 Links: 1 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 17:32:16.832550568 +0000 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 17:26:42.700445207 +0000 2023-10-03T17:32:31.467 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 17:26:42.700445207 +0000 2023-10-03T17:32:31.468 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T17:32:31.468 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T17:32:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T17:32:31.532 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T17:32:31.533 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000213844 s, 2.4 MB/s 2023-10-03T17:32:31.533 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T17:32:31.592 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-10-03T17:32:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T17:32:31.650 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 383995 Links: 1 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 17:32:16.832550568 +0000 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 17:26:43.789427590 +0000 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 17:26:43.789427590 +0000 2023-10-03T17:32:31.651 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T17:32:31.651 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T17:32:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T17:32:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T17:32:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.0002194 s, 2.3 MB/s 2023-10-03T17:32:31.717 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T17:32:31.775 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:32:31.776 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T17:32:31.776 INFO:tasks.ceph:Generating config... 2023-10-03T17:32:31.778 INFO:tasks.ceph:[client] rbd default features = 1 2023-10-03T17:32:31.778 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-10-03T17:32:31.778 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-10-03T17:32:31.778 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mgr] debug rbd = 20 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-03T17:32:31.778 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-03T17:32:31.778 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-03T17:32:31.779 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-03T17:32:31.780 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-10-03T17:32:31.780 INFO:tasks.ceph:Setting up mon.a... 2023-10-03T17:32:31.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-03T17:32:31.869 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.keyring 2023-10-03T17:32:31.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-03T17:32:31.918 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T17:32:31.947 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.31')] 2023-10-03T17:32:31.948 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug rbd': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd default features': 1}, 'mon.a': {}} 2023-10-03T17:32:31.948 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:31.948 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T17:32:32.005 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:32.088 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:32.088 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid 2a0c5350-120a-4bb6-a8cc-bda4017ada08 2023-10-03T17:32:32.088 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-10-03T17:32:32.088 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 2a0c5350-120a-4bb6-a8cc-bda4017ada08 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-10-03T17:32:32.090023+0000 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-10-03T17:32:32.090023+0000 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-10-03T17:32:32.089 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2023-10-03T17:32:32.090 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-03T17:32:32.148 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2a0c5350-120a-4bb6-a8cc-bda4017ada08... 2023-10-03T17:32:32.149 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-10-03T17:32:32.234 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.235 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-10-03T17:32:32.236 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.237 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on pool no app = False 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 2a0c5350-120a-4bb6-a8cc-bda4017ada08 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-10-03T17:32:32.238 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-10-03T17:32:32.239 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.239 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-10-03T17:32:32.239 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.240 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-10-03T17:32:32.247 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-10-03T17:32:32.247 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-10-03T17:32:32.247 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-10-03T17:32:32.248 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-10-03T17:32:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-10-03T17:32:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: bluefs allocator = hybrid 2023-10-03T17:32:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = hybrid 2023-10-03T17:32:32.249 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-10-03T17:32:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-10-03T17:32:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2023-10-03T17:32:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2023-10-03T17:32:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-03T17:32:32.251 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: osd shutdown pgref assert = True 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-03T17:32:32.252 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-10-03T17:32:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-10-03T17:32:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-03T17:32:32.253 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 20 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-10-03T17:32:32.255 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-10-03T17:32:32.256 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.260 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-10-03T17:32:32.261 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout: rbd default features = 1 2023-10-03T17:32:32.263 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-10-03T17:32:32.264 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-03T17:32:32.264 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-03T17:32:32.363 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-03T17:32:32.363 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:32.363 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-03T17:32:32.380 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:32.380 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-03T17:32:32.437 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T17:32:32.438 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:32.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-03T17:32:32.438 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-03T17:32:32.515 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:32.515 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:32.572 INFO:tasks.ceph:Setting up mon nodes... 2023-10-03T17:32:32.573 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-03T17:32:32.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T17:32:32.668 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-03T17:32:32.671 INFO:tasks.ceph:Setting up mds nodes... 2023-10-03T17:32:32.671 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-03T17:32:32.671 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-03T17:32:32.716 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-03T17:32:32.729 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-03T17:32:32.729 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2023-10-03T17:32:32.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-03T17:32:32.756 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:32:32.756 INFO:tasks.ceph:role: osd.0 2023-10-03T17:32:32.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T17:32:32.756 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-03T17:32:32.831 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:32:32.832 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:32:33.263 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-03T17:32:33.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-03T17:32:33.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-03T17:32:33.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-03T17:32:33.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-03T17:32:33.546 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:32:33.546 INFO:tasks.ceph:role: osd.1 2023-10-03T17:32:33.546 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T17:32:33.546 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-03T17:32:33.621 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:32:33.622 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:32:34.043 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-03T17:32:34.049 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-03T17:32:34.050 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-03T17:32:34.164 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-03T17:32:34.198 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-03T17:32:34.266 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-10-03T17:32:34.267 INFO:tasks.ceph:role: osd.2 2023-10-03T17:32:34.267 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T17:32:34.267 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-03T17:32:34.342 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-10-03T17:32:34.343 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-03T17:32:34.343 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-03T17:32:34.343 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-03T17:32:34.343 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-03T17:32:34.784 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-10-03T17:32:34.786 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-10-03T17:32:34.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-03T17:32:34.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-03T17:32:34.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:35.125 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.126+0000 7f88e17de640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-03T17:32:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.130+0000 7f88e17de640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-03T17:32:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.130+0000 7f88e17de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:32:35.129 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.130+0000 7f88e17de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:32:35.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.130+0000 7f88e17de640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-03T17:32:35.133 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:35.131+0000 7f88e17de640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-03T17:32:38.001 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-03T17:32:38.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.135+0000 7f81aaeb9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.136+0000 7f81aaeb9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.136+0000 7f81aaeb9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.136+0000 7f81aaeb9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.136+0000 7f81aaeb9640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-03T17:32:38.135 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:38.136+0000 7f81aaeb9640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-03T17:32:40.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-03T17:32:41.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.119+0000 7f8a7a10d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-03T17:32:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.119+0000 7f8a7a10d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-03T17:32:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.119+0000 7f8a7a10d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:32:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.119+0000 7f8a7a10d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:32:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.119+0000 7f8a7a10d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-03T17:32:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T17:32:41.120+0000 7f8a7a10d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-03T17:32:44.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-03T17:32:44.036 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-03T17:32:44.036 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:44.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-03T17:32:44.108 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:44.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-03T17:32:44.181 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:44.181 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-03T17:32:44.254 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:44.254 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-03T17:32:44.323 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:44.323 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-03T17:32:44.381 INFO:tasks.ceph:Adding keys to all mons... 2023-10-03T17:32:44.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAwUBxl/Xf4JxAAni0nXVJwJ/8TJpVxZ7pMPg== 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAzUBxl67CUBxAAtpxluzIr/3n9tO1sKBxMVA== 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-10-03T17:32:44.449 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQA2UBxllxcjCBAAPgweLjqfiwwk6fKR13Fj4Q== 2023-10-03T17:32:44.450 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-10-03T17:32:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQA5UBxlcVkkBxAAHP5KUgAQ64yLBoMdYesRIA== 2023-10-03T17:32:44.450 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-10-03T17:32:44.450 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQAwUBxlzEvOKhAACuFT7187ZNL0pUhDzTh0BA== 2023-10-03T17:32:44.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-03T17:32:44.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:32:44.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:32:44.655 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-03T17:32:44.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-03T17:32:44.769 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-03T17:32:44.770 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-03T17:32:44.797 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-03T17:32:44.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-03T17:32:44.924 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-03T17:32:44.982 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-03T17:32:44.982 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-03T17:32:44.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-03T17:32:45.024 INFO:tasks.ceph.mon.a:Started 2023-10-03T17:32:45.024 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-03T17:32:45.025 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-03T17:32:45.025 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-03T17:32:45.026 INFO:tasks.ceph.mgr.x:Started 2023-10-03T17:32:45.027 INFO:tasks.ceph:Setting crush tunables to default 2023-10-03T17:32:45.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd crush tunables default 2023-10-03T17:32:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-10-03T17:32:45.268 INFO:tasks.ceph:check_enable_crimson: False 2023-10-03T17:32:45.269 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-03T17:32:45.269 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:45.269 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-03T17:32:45.296 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:45.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-03T17:32:45.364 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:32:45.364 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-03T17:32:45.426 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:45.427+0000 7f586f140200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T17:32:45.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 96941377-c8f6-45de-8108-5751af5af033 0 2023-10-03T17:32:45.496 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:45.497+0000 7f586f140200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T17:32:45.577 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:45.578+0000 7f586f140200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T17:32:45.662 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:45.663+0000 7f586f140200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T17:32:45.689 INFO:teuthology.orchestra.run.smithi031.stdout:0 2023-10-03T17:32:45.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 2895e424-6c96-426c-802d-3b91e5cafcb3 1 2023-10-03T17:32:45.815 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:45.816+0000 7f586f140200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T17:32:45.961 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-10-03T17:32:45.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd new 58cf0e6f-2dfe-4737-8e92-0b5fc5950baf 2 2023-10-03T17:32:46.194 INFO:teuthology.orchestra.run.smithi031.stdout:2 2023-10-03T17:32:46.209 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-03T17:32:46.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-03T17:32:46.211 INFO:tasks.ceph.osd.0:Started 2023-10-03T17:32:46.212 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-03T17:32:46.212 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-03T17:32:46.214 INFO:tasks.ceph.osd.1:Started 2023-10-03T17:32:46.214 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-03T17:32:46.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-03T17:32:46.216 INFO:tasks.ceph.osd.2:Started 2023-10-03T17:32:46.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:32:46.233 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:46.234+0000 7f586f140200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T17:32:46.371 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:46.372+0000 7f586f140200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T17:32:46.434 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:32:46.435 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:32:46.195093+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:32:46.447 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-03T17:32:46.447 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-03T17:32:46.463 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:46.464+0000 7f586f140200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T17:32:46.559 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:46.561+0000 7f586f140200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T17:32:46.750 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:32:46.824 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:32:46.825+0000 7f063c5f9640 -1 Falling back to public interface 2023-10-03T17:32:46.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:32:46.842+0000 7fc52e6d4640 -1 Falling back to public interface 2023-10-03T17:32:46.849 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:32:46.850+0000 7f9f3a65c640 -1 Falling back to public interface 2023-10-03T17:32:46.962 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-03T17:32:46.963 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":5,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:32:46.195093+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:32:46.974 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T17:32:47.043 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:47.044+0000 7f586f140200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T17:32:47.781 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:47.782+0000 7f586f140200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-03T17:32:47.853 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:47.854+0000 7f586f140200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T17:32:48.578 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:48.579+0000 7f586f140200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T17:32:48.660 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:48.661+0000 7f586f140200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T17:32:48.807 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:48.808+0000 7f586f140200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T17:32:49.054 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.055+0000 7f586f140200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-03T17:32:49.295 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.296+0000 7f586f140200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T17:32:49.436 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.437+0000 7f586f140200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T17:32:49.689 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.690+0000 7f586f140200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T17:32:49.854 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.855+0000 7f586f140200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T17:32:49.975 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:49.976+0000 7f586f140200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T17:32:50.316 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:50.317+0000 7f586f140200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T17:32:50.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:32:50.328+0000 7fc52e6d4640 -1 osd.1 0 log_to_monitors true 2023-10-03T17:32:50.345 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:32:50.346+0000 7f063c5f9640 -1 osd.0 0 log_to_monitors true 2023-10-03T17:32:50.383 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:32:50.384+0000 7f9f3a65c640 -1 osd.2 0 log_to_monitors true 2023-10-03T17:32:50.431 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:50.432+0000 7f586f140200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T17:32:50.627 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:50.628+0000 7f586f140200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T17:32:51.253 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:51.253+0000 7f586f140200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T17:32:52.349 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:32:52.349+0000 7f58270d6700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-10-03T17:32:53.277 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:32:53.824 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-03T17:32:53.824 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":7,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:32:52.125053+0000","last_up_change":"0.000000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:32:53.843 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-10-03T17:32:53.877 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:32:53.877+0000 7f9f2ea14700 -1 osd.2 0 waiting for initial osdmap 2023-10-03T17:32:53.892 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:32:53.892+0000 7f9f265ee700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:32:53.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:32:53.930+0000 7fc522a8c700 -1 osd.1 0 waiting for initial osdmap 2023-10-03T17:32:53.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:32:53.943+0000 7fc51a666700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:32:53.945 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:32:53.945+0000 7f06309b1700 -1 osd.0 0 waiting for initial osdmap 2023-10-03T17:32:53.961 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:32:53.961+0000 7f062858b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T17:33:00.145 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-03T17:33:00.487 INFO:teuthology.misc.health.smithi031.stdout: 2023-10-03T17:33:00.488 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":11,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:32:58.184876+0000","last_up_change":"2023-10-03T17:32:54.879734+0000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:32:55.331164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6801","nonce":3351865178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6803","nonce":3351865178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6807","nonce":3351865178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6805","nonce":3351865178}]},"public_addr":"172.21.15.31:6801/3351865178","cluster_addr":"172.21.15.31:6803/3351865178","heartbeat_back_addr":"172.21.15.31:6807/3351865178","heartbeat_front_addr":"172.21.15.31:6805/3351865178","state":["exists","up"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6809","nonce":875549188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6811","nonce":875549188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6815","nonce":875549188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6813","nonce":875549188}]},"public_addr":"172.21.15.31:6809/875549188","cluster_addr":"172.21.15.31:6811/875549188","heartbeat_back_addr":"172.21.15.31:6815/875549188","heartbeat_front_addr":"172.21.15.31:6813/875549188","state":["exists","up"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6817","nonce":2591775043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6819","nonce":2591775043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6823","nonce":2591775043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6821","nonce":2591775043}]},"public_addr":"172.21.15.31:6817/2591775043","cluster_addr":"172.21.15.31:6819/2591775043","heartbeat_back_addr":"172.21.15.31:6823/2591775043","heartbeat_front_addr":"172.21.15.31:6821/2591775043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.335718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.378460+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.390043+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:33:00.500 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-10-03T17:33:00.500 INFO:tasks.ceph:Creating RBD pool 2023-10-03T17:33:00.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-03T17:33:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-10-03T17:33:01.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-03T17:33:02.214 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-03T17:33:02.232 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-03T17:33:02.233 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-03T17:33:02.233 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-03T17:33:02.233 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-03T17:33:02.233 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-03T17:33:02.633 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:02.646 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":497198010},{"type":"v1","addr":"172.21.15.31:6825","nonce":497198010}]},"active_addr":"172.21.15.31:6825/497198010","active_change":"2023-10-03T17:32:51.267758+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2817142752}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1062116524}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2220906593}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":883332448}]}]} 2023-10-03T17:33:02.649 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-03T17:33:02.649 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-03T17:33:02.649 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:33:02.984 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:02.984 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:33:02.206349+0000","last_up_change":"2023-10-03T17:32:54.879734+0000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:32:55.331164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:33:00.866205+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6801","nonce":3351865178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6803","nonce":3351865178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6807","nonce":3351865178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6805","nonce":3351865178}]},"public_addr":"172.21.15.31:6801/3351865178","cluster_addr":"172.21.15.31:6803/3351865178","heartbeat_back_addr":"172.21.15.31:6807/3351865178","heartbeat_front_addr":"172.21.15.31:6805/3351865178","state":["exists","up"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6809","nonce":875549188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6811","nonce":875549188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6815","nonce":875549188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6813","nonce":875549188}]},"public_addr":"172.21.15.31:6809/875549188","cluster_addr":"172.21.15.31:6811/875549188","heartbeat_back_addr":"172.21.15.31:6815/875549188","heartbeat_front_addr":"172.21.15.31:6813/875549188","state":["exists","up"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6817","nonce":2591775043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6819","nonce":2591775043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6823","nonce":2591775043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6821","nonce":2591775043}]},"public_addr":"172.21.15.31:6817/2591775043","cluster_addr":"172.21.15.31:6819/2591775043","heartbeat_back_addr":"172.21.15.31:6823/2591775043","heartbeat_front_addr":"172.21.15.31:6821/2591775043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.335718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.378460+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.390043+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:33:02.998 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-03T17:33:02.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T17:33:03.335 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:03.335 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T17:33:03.212180+0000","last_up_change":"2023-10-03T17:32:54.879734+0000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:32:55.331164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:33:00.866205+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6801","nonce":3351865178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6803","nonce":3351865178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6807","nonce":3351865178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6805","nonce":3351865178}]},"public_addr":"172.21.15.31:6801/3351865178","cluster_addr":"172.21.15.31:6803/3351865178","heartbeat_back_addr":"172.21.15.31:6807/3351865178","heartbeat_front_addr":"172.21.15.31:6805/3351865178","state":["exists","up"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6809","nonce":875549188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6811","nonce":875549188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6815","nonce":875549188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6813","nonce":875549188}]},"public_addr":"172.21.15.31:6809/875549188","cluster_addr":"172.21.15.31:6811/875549188","heartbeat_back_addr":"172.21.15.31:6815/875549188","heartbeat_front_addr":"172.21.15.31:6813/875549188","state":["exists","up"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6817","nonce":2591775043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6819","nonce":2591775043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6823","nonce":2591775043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6821","nonce":2591775043}]},"public_addr":"172.21.15.31:6817/2591775043","cluster_addr":"172.21.15.31:6819/2591775043","heartbeat_back_addr":"172.21.15.31:6823/2591775043","heartbeat_front_addr":"172.21.15.31:6821/2591775043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.335718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.378460+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.390043+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T17:33:03.349 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-03T17:33:03.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-03T17:33:03.350 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-03T17:33:03.519 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:03.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T17:33:03.525 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:03.526 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T17:33:03.526 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:03.527 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T17:33:03.865 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:03.880 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-03T17:33:03.914 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:03.915 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:03.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-03T17:33:03.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-03T17:33:04.881 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T17:33:04.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T17:33:04.932 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T17:33:05.253 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:05.267 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.0 2023-10-03T17:33:05.282 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:05.292 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-10-03T17:33:05.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.2 2023-10-03T17:33:05.307 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738372 for osd.1 2023-10-03T17:33:06.268 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-03T17:33:06.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-03T17:33:06.308 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-03T17:33:06.649 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:06.664 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.0 2023-10-03T17:33:06.664 DEBUG:teuthology.parallel:result is None 2023-10-03T17:33:06.669 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:06.683 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-10-03T17:33:06.684 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.2 2023-10-03T17:33:06.684 DEBUG:teuthology.parallel:result is None 2023-10-03T17:33:06.697 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738373 got 34359738373 for osd.1 2023-10-03T17:33:06.697 DEBUG:teuthology.parallel:result is None 2023-10-03T17:33:06.697 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T17:33:06.697 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:33:07.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:07.071 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T17:33:07.083 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-03T17:33:05.276167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":11,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.067391"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.228267+0000","last_change":"2023-10-03T17:33:02.224277+0000","last_active":"2023-10-03T17:33:02.228267+0000","last_peered":"2023-10-03T17:33:02.228267+0000","last_clean":"2023-10-03T17:33:02.228267+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T17:33:02.228267+0000","last_undegraded":"2023-10-03T17:33:02.228267+0000","last_fullsized":"2023-10-03T17:33:02.228267+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:37:54.293196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.228284+0000","last_change":"2023-10-03T17:33:02.224408+0000","last_active":"2023-10-03T17:33:02.228284+0000","last_peered":"2023-10-03T17:33:02.228284+0000","last_clean":"2023-10-03T17:33:02.228284+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T17:33:02.228284+0000","last_undegraded":"2023-10-03T17:33:02.228284+0000","last_fullsized":"2023-10-03T17:33:02.228284+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:04:42.580745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.231182+0000","last_change":"2023-10-03T17:33:02.226636+0000","last_active":"2023-10-03T17:33:02.231182+0000","last_peered":"2023-10-03T17:33:02.231182+0000","last_clean":"2023-10-03T17:33:02.231182+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T17:33:02.231182+0000","last_undegraded":"2023-10-03T17:33:02.231182+0000","last_fullsized":"2023-10-03T17:33:02.231182+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:48:12.539933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.231157+0000","last_change":"2023-10-03T17:33:02.226645+0000","last_active":"2023-10-03T17:33:02.231157+0000","last_peered":"2023-10-03T17:33:02.231157+0000","last_clean":"2023-10-03T17:33:02.231157+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T17:33:02.231157+0000","last_undegraded":"2023-10-03T17:33:02.231157+0000","last_fullsized":"2023-10-03T17:33:02.231157+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:32:05.860877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-10-03T17:33:01.210275+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T17:33:01.210275+0000","last_peered":"2023-10-03T17:33:01.210275+0000","last_clean":"2023-10-03T17:33:01.210275+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T17:33:01.210275+0000","last_undegraded":"2023-10-03T17:33:01.210275+0000","last_fullsized":"2023-10-03T17:33:01.210275+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.228358+0000","last_change":"2023-10-03T17:33:02.224022+0000","last_active":"2023-10-03T17:33:02.228358+0000","last_peered":"2023-10-03T17:33:02.228358+0000","last_clean":"2023-10-03T17:33:02.228358+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T17:33:02.228358+0000","last_undegraded":"2023-10-03T17:33:02.228358+0000","last_fullsized":"2023-10-03T17:33:02.228358+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:33:49.557367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.231177+0000","last_change":"2023-10-03T17:33:02.226719+0000","last_active":"2023-10-03T17:33:02.231177+0000","last_peered":"2023-10-03T17:33:02.231177+0000","last_clean":"2023-10-03T17:33:02.231177+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T17:33:02.231177+0000","last_undegraded":"2023-10-03T17:33:02.231177+0000","last_fullsized":"2023-10-03T17:33:02.231177+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:37:31.661862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.214430+0000","last_change":"2023-10-03T17:33:02.226724+0000","last_active":"2023-10-03T17:33:03.214430+0000","last_peered":"2023-10-03T17:33:03.214430+0000","last_clean":"2023-10-03T17:33:03.214430+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T17:33:03.214430+0000","last_undegraded":"2023-10-03T17:33:03.214430+0000","last_fullsized":"2023-10-03T17:33:03.214430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:32.697821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.228325+0000","last_change":"2023-10-03T17:33:02.224305+0000","last_active":"2023-10-03T17:33:02.228325+0000","last_peered":"2023-10-03T17:33:02.228325+0000","last_clean":"2023-10-03T17:33:02.228325+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T17:33:02.228325+0000","last_undegraded":"2023-10-03T17:33:02.228325+0000","last_fullsized":"2023-10-03T17:33:02.228325+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:27:20.389014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20899999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:33:07.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T17:33:07.419 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:07.422 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T17:33:07.433 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-10-03T17:33:07.276488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82588,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032932,"statfs":{"total":289910292480,"available":289825722368,"internally_reserved":0,"allocated":1556480,"data_stored":1285127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4781,"internal_metadata":82963795},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.067711"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751520+0000","last_change":"2023-10-03T17:33:02.224277+0000","last_active":"2023-10-03T17:33:03.751520+0000","last_peered":"2023-10-03T17:33:03.751520+0000","last_clean":"2023-10-03T17:33:03.751520+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T17:33:03.751520+0000","last_undegraded":"2023-10-03T17:33:03.751520+0000","last_fullsized":"2023-10-03T17:33:03.751520+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:25:18.045725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751326+0000","last_change":"2023-10-03T17:33:02.224408+0000","last_active":"2023-10-03T17:33:03.751326+0000","last_peered":"2023-10-03T17:33:03.751326+0000","last_clean":"2023-10-03T17:33:03.751326+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T17:33:03.751326+0000","last_undegraded":"2023-10-03T17:33:03.751326+0000","last_fullsized":"2023-10-03T17:33:03.751326+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:52:46.492068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751580+0000","last_change":"2023-10-03T17:33:02.226636+0000","last_active":"2023-10-03T17:33:03.751580+0000","last_peered":"2023-10-03T17:33:03.751580+0000","last_clean":"2023-10-03T17:33:03.751580+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T17:33:03.751580+0000","last_undegraded":"2023-10-03T17:33:03.751580+0000","last_fullsized":"2023-10-03T17:33:03.751580+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:51:10.246275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751198+0000","last_change":"2023-10-03T17:33:02.226645+0000","last_active":"2023-10-03T17:33:03.751198+0000","last_peered":"2023-10-03T17:33:03.751198+0000","last_clean":"2023-10-03T17:33:03.751198+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T17:33:03.751198+0000","last_undegraded":"2023-10-03T17:33:03.751198+0000","last_fullsized":"2023-10-03T17:33:03.751198+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:08:20.982668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":141,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-10-03T17:33:02.216906+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T17:33:02.216906+0000","last_peered":"2023-10-03T17:33:02.216906+0000","last_clean":"2023-10-03T17:33:02.216906+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T17:33:02.216906+0000","last_undegraded":"2023-10-03T17:33:02.216906+0000","last_fullsized":"2023-10-03T17:33:02.216906+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751325+0000","last_change":"2023-10-03T17:33:02.224022+0000","last_active":"2023-10-03T17:33:03.751325+0000","last_peered":"2023-10-03T17:33:03.751325+0000","last_clean":"2023-10-03T17:33:03.751325+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T17:33:03.751325+0000","last_undegraded":"2023-10-03T17:33:03.751325+0000","last_fullsized":"2023-10-03T17:33:03.751325+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:25:07.485321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751324+0000","last_change":"2023-10-03T17:33:02.226719+0000","last_active":"2023-10-03T17:33:03.751324+0000","last_peered":"2023-10-03T17:33:03.751324+0000","last_clean":"2023-10-03T17:33:03.751324+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T17:33:03.751324+0000","last_undegraded":"2023-10-03T17:33:03.751324+0000","last_fullsized":"2023-10-03T17:33:03.751324+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:14:56.026430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.214430+0000","last_change":"2023-10-03T17:33:02.226724+0000","last_active":"2023-10-03T17:33:03.214430+0000","last_peered":"2023-10-03T17:33:03.214430+0000","last_clean":"2023-10-03T17:33:03.214430+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T17:33:03.214430+0000","last_undegraded":"2023-10-03T17:33:03.214430+0000","last_fullsized":"2023-10-03T17:33:03.214430+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:01:32.697821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-10-03T17:33:03.751499+0000","last_change":"2023-10-03T17:33:02.224305+0000","last_active":"2023-10-03T17:33:03.751499+0000","last_peered":"2023-10-03T17:33:03.751499+0000","last_clean":"2023-10-03T17:33:03.751499+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T17:33:03.751499+0000","last_undegraded":"2023-10-03T17:33:03.751499+0000","last_fullsized":"2023-10-03T17:33:03.751499+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:19:06.805553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":34797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T17:33:07.435 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T17:33:07.435 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-03T17:33:07.435 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-03T17:33:07.435 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-03T17:33:07.813 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:07.813 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-03T17:33:07.827 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-03T17:33:07.827 INFO:teuthology.run_tasks:Running task install... 2023-10-03T17:33:07.836 DEBUG:teuthology.task.install:project ceph 2023-10-03T17:33:07.837 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'}} 2023-10-03T17:33:07.837 DEBUG:teuthology.task.install:config {'extra_system_packages': ['fio'], 'flavor': 'default', 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'} 2023-10-03T17:33:07.837 INFO:teuthology.task.install:Using flavor: default 2023-10-03T17:33:07.842 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-03T17:33:07.843 INFO:teuthology.task.install:extra packages: [] 2023-10-03T17:33:07.843 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': ['fio'], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad', 'tag': None, 'wait_for_package': False} 2023-10-03T17:33:07.843 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:33:08.076 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/ 2023-10-03T17:33:08.076 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6336.ga7de91bf 2023-10-03T17:33:08.184 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-ajarr-63009-only-reacquire-kickstart/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-03T17:33:08.184 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:33:08.184 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-03T17:33:08.215 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, fio, fio on remote rpm x86_64 2023-10-03T17:33:08.215 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-03T17:33:08.285 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-03T17:33:08.355 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-03T17:33:08.412 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-10-03T17:33:08.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-10-03T17:33:08.843 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:33:09.080 INFO:teuthology.orchestra.run.smithi031.stdout:53 files removed 2023-10-03T17:33:09.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd fio fio 2023-10-03T17:33:09.506 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-10-03T17:33:10.348 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 175 kB/s | 81 kB 00:00 2023-10-03T17:33:10.673 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 33 kB/s | 9.4 kB 00:00 2023-10-03T17:33:10.916 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 6.8 kB/s | 1.5 kB 00:00 2023-10-03T17:33:11.050 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 31 kB/s | 3.5 kB 00:00 2023-10-03T17:33:12.534 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-10-03T17:33:16.301 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 473 kB/s | 24 kB 00:00 2023-10-03T17:33:17.269 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 53 MB 00:00 2023-10-03T17:33:26.146 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 47 MB 00:00 2023-10-03T17:33:38.005 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-radosgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.006 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-test-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.007 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.008 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.010 INFO:teuthology.orchestra.run.smithi031.stdout:Package cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.011 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-immutable-object-cache-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.012 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.014 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-dashboard-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.015 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-diskprediction-local-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.016 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-rook-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.017 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-cephadm-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.024 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.024 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-volume-2:18.0.0-6336.ga7de91bf.el8.noarch is already installed. 2023-10-03T17:33:38.024 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.024 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.024 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs-devel-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.025 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.026 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.028 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.029 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.030 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.032 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.033 INFO:teuthology.orchestra.run.smithi031.stdout:Package rbd-fuse-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.034 INFO:teuthology.orchestra.run.smithi031.stdout:Package rbd-mirror-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.036 INFO:teuthology.orchestra.run.smithi031.stdout:Package rbd-nbd-2:18.0.0-6336.ga7de91bf.el8.x86_64 is already installed. 2023-10-03T17:33:38.155 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T17:33:38.156 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:33:38.156 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T17:33:38.156 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: fio x86_64 3.19-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: daxctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 42 k 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemblk x86_64 1.6.1-1.el8 rhel-8-for-x86_64-appstream-rpms 92 k 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: ndctl-libs x86_64 71.1-3.el8 rhel-8-for-x86_64-baseos-rpms 79 k 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout:Install 4 Packages 2023-10-03T17:33:38.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:38.158 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 750 k 2023-10-03T17:33:38.158 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 2.5 M 2023-10-03T17:33:38.158 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T17:33:38.230 INFO:teuthology.orchestra.run.smithi031.stdout:(1/4): daxctl-libs-71.1-3.el8.x86_64.rpm 588 kB/s | 42 kB 00:00 2023-10-03T17:33:38.254 INFO:teuthology.orchestra.run.smithi031.stdout:(2/4): ndctl-libs-71.1-3.el8.x86_64.rpm 819 kB/s | 79 kB 00:00 2023-10-03T17:33:38.280 INFO:teuthology.orchestra.run.smithi031.stdout:(3/4): libpmemblk-1.6.1-1.el8.x86_64.rpm 1.8 MB/s | 92 kB 00:00 2023-10-03T17:33:38.305 INFO:teuthology.orchestra.run.smithi031.stdout:(4/4): fio-3.19-3.el8.x86_64.rpm 3.6 MB/s | 538 kB 00:00 2023-10-03T17:33:38.305 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T17:33:38.306 INFO:teuthology.orchestra.run.smithi031.stdout:Total 5.0 MB/s | 750 kB 00:00 2023-10-03T17:33:38.348 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T17:33:38.361 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T17:33:38.362 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T17:33:38.443 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T17:33:38.447 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T17:33:38.658 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T17:33:39.055 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : daxctl-libs-71.1-3.el8.x86_64 1/4 2023-10-03T17:33:39.134 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: daxctl-libs-71.1-3.el8.x86_64 1/4 2023-10-03T17:33:39.330 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ndctl-libs-71.1-3.el8.x86_64 2/4 2023-10-03T17:33:39.401 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ndctl-libs-71.1-3.el8.x86_64 2/4 2023-10-03T17:33:39.588 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-10-03T17:33:39.721 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libpmemblk-1.6.1-1.el8.x86_64 3/4 2023-10-03T17:33:39.903 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fio-3.19-3.el8.x86_64 4/4 2023-10-03T17:33:40.147 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: fio-3.19-3.el8.x86_64 4/4 2023-10-03T17:33:40.148 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : daxctl-libs-71.1-3.el8.x86_64 1/4 2023-10-03T17:33:40.148 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ndctl-libs-71.1-3.el8.x86_64 2/4 2023-10-03T17:33:40.148 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fio-3.19-3.el8.x86_64 3/4 2023-10-03T17:33:41.281 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemblk-1.6.1-1.el8.x86_64 4/4 2023-10-03T17:33:41.281 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:16 ago on Tue 03 Oct 2023 05:33:25 PM UTC. 2023-10-03T17:33:43.769 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout: daxctl-libs-71.1-3.el8.x86_64 fio-3.19-3.el8.x86_64 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemblk-1.6.1-1.el8.x86_64 ndctl-libs-71.1-3.el8.x86_64 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T17:33:43.969 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T17:33:44.100 DEBUG:teuthology.parallel:result is None 2023-10-03T17:33:44.101 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:33:44.244 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-03T17:33:44.268 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-6336.ga7de91bf.el8 2023-10-03T17:33:44.268 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6336.ga7de91bf.el8 2023-10-03T17:33:44.269 INFO:teuthology.task.install:The correct ceph version 18.0.0-6336.ga7de91bf is installed. 2023-10-03T17:33:44.269 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2023-10-03T17:33:44.269 INFO:teuthology.run_tasks:Running task workunit... 2023-10-03T17:33:44.297 INFO:tasks.workunit:Pulling workunits from ref a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:33:44.297 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-03T17:33:44.297 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-03T17:33:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T17:33:44.325 INFO:teuthology.orchestra.run.smithi031.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-03T17:33:44.326 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-03T17:33:44.383 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-03T17:33:44.385 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-03T17:33:44.440 INFO:tasks.workunit:timeout=3h 2023-10-03T17:33:44.440 INFO:tasks.workunit:cleanup=True 2023-10-03T17:33:44.440 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a7de91bfa5778a0d559a7e9bc8e35f93456b41ad 2023-10-03T17:33:44.497 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-03T17:34:45.241 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 99% (12012/12041) Updating files: 100% (12041/12041) Updating files: 100% (12041/12041), done. 2023-10-03T17:34:45.595 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to 'a7de91bfa5778a0d559a7e9bc8e35f93456b41ad'. 2023-10-03T17:34:45.595 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.595 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.596 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-03T17:34:45.597 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T17:34:45.597 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at a7de91bfa57 librbd: fix another block when waiting for exclusive lock 2023-10-03T17:34:45.598 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-03T17:34:45.662 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-03T17:34:45.664 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T17:34:45.664 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-03T17:34:46.221 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-03T17:34:46.273 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-03T17:34:46.312 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T17:34:46.313 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T17:34:46.313 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-03T17:34:46.351 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T17:34:46.354 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T17:34:46.355 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-03T17:34:46.412 INFO:tasks.workunit:Running workunits matching rbd/rbd_support_module_recovery.sh on client.0... 2023-10-03T17:34:46.413 INFO:tasks.workunit:Running workunit rbd/rbd_support_module_recovery.sh... 2023-10-03T17:34:46.414 DEBUG:teuthology.orchestra.run.smithi031:workunit test rbd/rbd_support_module_recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh 2023-10-03T17:34:46.473 INFO:tasks.workunit.client.0.smithi031.stderr:+ POOL=rbd 2023-10-03T17:34:46.473 INFO:tasks.workunit.client.0.smithi031.stderr:+ IMAGE_PREFIX=image 2023-10-03T17:34:46.473 INFO:tasks.workunit.client.0.smithi031.stderr:+ NUM_IMAGES=20 2023-10-03T17:34:46.473 INFO:tasks.workunit.client.0.smithi031.stderr:+ RUN_TIME=3600 2023-10-03T17:34:46.474 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror pool enable rbd image 2023-10-03T17:34:46.527 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror pool peer add rbd dummy 2023-10-03T17:34:46.562 INFO:tasks.workunit.client.0.smithi031.stdout:0c3abd0f-a859-439f-9d69-86406496e561 2023-10-03T17:34:46.565 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i = 1 )) 2023-10-03T17:34:46.565 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:46.565 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image1 2023-10-03T17:34:48.154 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image1 snapshot 2023-10-03T17:34:49.155 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:49.160 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image1 1m 2023-10-03T17:34:49.215 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:49.215 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:49.215 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image2 2023-10-03T17:34:49.257 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image2 snapshot 2023-10-03T17:34:50.158 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:50.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image2 1m 2023-10-03T17:34:50.219 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:50.219 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:50.219 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image3 2023-10-03T17:34:50.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image3 snapshot 2023-10-03T17:34:51.166 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:51.172 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image3 1m 2023-10-03T17:34:51.226 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:51.226 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:51.226 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image4 2023-10-03T17:34:51.274 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image4 snapshot 2023-10-03T17:34:52.176 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:52.181 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image4 1m 2023-10-03T17:34:52.237 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:52.237 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:52.237 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image5 2023-10-03T17:34:52.282 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image5 snapshot 2023-10-03T17:34:53.174 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:53.180 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image5 1m 2023-10-03T17:34:53.234 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:53.235 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:53.235 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image6 2023-10-03T17:34:53.278 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image6 snapshot 2023-10-03T17:34:54.185 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:54.191 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image6 1m 2023-10-03T17:34:54.245 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:54.245 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:54.246 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image7 2023-10-03T17:34:54.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image7 snapshot 2023-10-03T17:34:55.151 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:55.156 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image7 1m 2023-10-03T17:34:55.210 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:55.211 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:55.211 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image8 2023-10-03T17:34:55.257 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image8 snapshot 2023-10-03T17:34:56.188 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:56.194 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image8 1m 2023-10-03T17:34:56.248 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:56.248 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:56.248 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image9 2023-10-03T17:34:56.295 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image9 snapshot 2023-10-03T17:34:57.192 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:57.198 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image9 1m 2023-10-03T17:34:57.254 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:57.254 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:57.254 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image10 2023-10-03T17:34:57.300 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image10 snapshot 2023-10-03T17:34:58.209 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:58.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image10 1m 2023-10-03T17:34:58.268 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:58.268 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:58.268 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image11 2023-10-03T17:34:58.314 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image11 snapshot 2023-10-03T17:34:59.209 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:34:59.215 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image11 1m 2023-10-03T17:34:59.269 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:34:59.269 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:34:59.269 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image12 2023-10-03T17:34:59.314 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image12 snapshot 2023-10-03T17:35:00.224 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:00.231 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image12 1m 2023-10-03T17:35:00.284 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:00.284 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:00.284 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image13 2023-10-03T17:35:00.327 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image13 snapshot 2023-10-03T17:35:01.220 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:01.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image13 1m 2023-10-03T17:35:01.280 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:01.280 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:01.280 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image14 2023-10-03T17:35:01.326 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image14 snapshot 2023-10-03T17:35:02.230 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:02.236 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image14 1m 2023-10-03T17:35:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:02.291 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image15 2023-10-03T17:35:02.337 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image15 snapshot 2023-10-03T17:35:03.237 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:03.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image15 1m 2023-10-03T17:35:03.296 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:03.296 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:03.296 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image16 2023-10-03T17:35:03.544 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image16 snapshot 2023-10-03T17:35:04.246 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:04.251 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image16 1m 2023-10-03T17:35:04.307 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:04.307 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:04.307 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image17 2023-10-03T17:35:04.351 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image17 snapshot 2023-10-03T17:35:05.245 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:05.251 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image17 1m 2023-10-03T17:35:05.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:05.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:05.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image18 2023-10-03T17:35:05.356 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image18 snapshot 2023-10-03T17:35:06.254 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:06.259 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image18 1m 2023-10-03T17:35:06.314 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:06.314 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:06.314 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image19 2023-10-03T17:35:06.360 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image19 snapshot 2023-10-03T17:35:07.258 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:07.263 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image19 1m 2023-10-03T17:35:07.317 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:07.317 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:07.317 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd create --size 1G rbd/image20 2023-10-03T17:35:07.364 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror image enable rbd/image20 snapshot 2023-10-03T17:35:08.261 INFO:tasks.workunit.client.0.smithi031.stdout:Mirroring enabled 2023-10-03T17:35:08.267 INFO:tasks.workunit.client.0.smithi031.stderr:+ rbd mirror snapshot schedule add -p rbd --image image20 1m 2023-10-03T17:35:08.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i = 1 )) 2023-10-03T17:35:08.322 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.322 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image1 2023-10-03T17:35:08.497 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd0 2023-10-03T17:35:08.497 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.497 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.497 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.497 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image2 2023-10-03T17:35:08.564 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd1 2023-10-03T17:35:08.564 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.564 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.564 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd1 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.564 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image3 2023-10-03T17:35:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd2 2023-10-03T17:35:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd2 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.639 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image4 2023-10-03T17:35:08.715 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd3 2023-10-03T17:35:08.715 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.716 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.716 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd3 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.716 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image5 2023-10-03T17:35:08.838 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd4 2023-10-03T17:35:08.838 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd4 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.839 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image6 2023-10-03T17:35:08.968 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd5 2023-10-03T17:35:08.968 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:08.968 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:08.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd5 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:08.969 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image7 2023-10-03T17:35:09.135 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd6 2023-10-03T17:35:09.136 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:09.136 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:09.136 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd6 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:09.136 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image8 2023-10-03T17:35:09.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd7 2023-10-03T17:35:09.345 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:09.345 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:09.345 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image9 2023-10-03T17:35:09.345 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd7 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:09.599 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd8 2023-10-03T17:35:09.599 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:09.599 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:09.603 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd8 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:09.603 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image10 2023-10-03T17:35:09.794 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd9 2023-10-03T17:35:09.795 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:09.795 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:09.795 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd9 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:09.814 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image11 2023-10-03T17:35:09.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd10 2023-10-03T17:35:09.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:09.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:09.989 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image12 2023-10-03T17:35:09.998 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd10 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:10.258 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd11 2023-10-03T17:35:10.258 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:10.258 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:10.258 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd11 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:10.258 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image13 2023-10-03T17:35:10.542 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd12 2023-10-03T17:35:10.542 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:10.542 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:10.543 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd12 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:10.554 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image14 2023-10-03T17:35:10.963 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd13 2023-10-03T17:35:10.963 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:10.963 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:10.963 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd13 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:10.978 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image15 2023-10-03T17:35:11.199 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd14 2023-10-03T17:35:11.199 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:11.199 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:11.202 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd14 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:11.209 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image16 2023-10-03T17:35:11.528 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd15 2023-10-03T17:35:11.528 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:11.528 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:11.548 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd15 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:11.560 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image17 2023-10-03T17:35:11.874 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd16 2023-10-03T17:35:11.874 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:11.874 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:11.880 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd16 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:11.883 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image18 2023-10-03T17:35:12.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd17 2023-10-03T17:35:12.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:12.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:12.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd17 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:12.254 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image19 2023-10-03T17:35:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd18 2023-10-03T17:35:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:12.579 INFO:tasks.workunit.client.0.smithi031.stderr:++ sudo rbd device map rbd/image20 2023-10-03T17:35:12.589 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd18 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:12.940 INFO:tasks.workunit.client.0.smithi031.stderr:+ DEVS[$i]=/dev/rbd19 2023-10-03T17:35:12.940 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i++ )) 2023-10-03T17:35:12.941 INFO:tasks.workunit.client.0.smithi031.stderr:+ (( i <= 20 )) 2023-10-03T17:35:12.947 INFO:tasks.workunit.client.0.smithi031.stderr:+ fio --name=fiotest --filename=/dev/rbd19 --rw=randrw --bs=4K --direct=1 --ioengine=libaio --iodepth=2 --runtime=43200 --time_based 2023-10-03T17:35:12.956 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354512 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ END_TIME=1696358112 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR= 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR= 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354512 -le 1696358112 ]] 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ '' != '' ]] 2023-10-03T17:35:12.957 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:35:22.973 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:35:22.979 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:35:22.979 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:35:22.986 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:35:24.228 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2220906593 2023-10-03T17:35:24.235 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:35:24.235 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354524 2023-10-03T17:35:24.236 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354524 -le 1696358112 ]] 2023-10-03T17:35:24.236 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2220906593 != '' ]] 2023-10-03T17:35:24.236 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2220906593 2023-10-03T17:35:26.435 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2220906593 until 2023-10-03T18:35:25.719389+0000 (3600 sec) 2023-10-03T17:35:26.462 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2220906593 2023-10-03T17:35:26.462 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:35:27.460 INFO:tasks.workunit.client.0.smithi031.stderr:listed 1 entries 2023-10-03T17:35:27.480 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2220906593 2023-10-03T17:35:27.480 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:35:37.482 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:35:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:35:37.490 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:35:37.503 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:35:38.713 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2220906593 2023-10-03T17:35:38.713 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:35:38.714 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354538 2023-10-03T17:35:38.714 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354538 -le 1696358112 ]] 2023-10-03T17:35:38.715 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2220906593 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\2\2\0\9\0\6\5\9\3 ]] 2023-10-03T17:35:38.715 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:35:48.719 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:35:48.719 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:35:48.719 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:35:48.719 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:35:49.973 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2220906593 2023-10-03T17:35:49.973 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:35:49.974 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354549 2023-10-03T17:35:49.974 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354549 -le 1696358112 ]] 2023-10-03T17:35:49.974 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2220906593 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\2\2\0\9\0\6\5\9\3 ]] 2023-10-03T17:35:49.974 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:35:59.977 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:35:59.977 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:35:59.977 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:35:59.978 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:01.160 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2220906593 2023-10-03T17:36:01.160 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:36:01.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354561 2023-10-03T17:36:01.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354561 -le 1696358112 ]] 2023-10-03T17:36:01.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2220906593 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\2\2\0\9\0\6\5\9\3 ]] 2023-10-03T17:36:01.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:36:11.172 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:36:11.177 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:11.181 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:11.188 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:11.684 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.679+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:36:11.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.694+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:36:11.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.694+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:36:11.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.694+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:36:11.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.694+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:36:11.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.695+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:36:11.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.695+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:36:11.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.703+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:36:11.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.705+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:36:11.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:11.748+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:36:12.357 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:36:12.358 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:36:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354572 2023-10-03T17:36:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354572 -le 1696358112 ]] 2023-10-03T17:36:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\2\2\0\9\0\6\5\9\3 ]] 2023-10-03T17:36:12.359 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2110891374 2023-10-03T17:36:14.488 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2110891374 until 2023-10-03T18:36:13.774834+0000 (3600 sec) 2023-10-03T17:36:14.503 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2110891374 2023-10-03T17:36:14.504 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:36:15.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.095+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:36:15.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.110+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:36:15.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.179+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:36:15.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.187+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:36:15.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.187+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:36:15.214 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.204+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:36:15.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:36:15.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:36:15.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:36:15.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:36:15.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:36:15.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:15.432+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:36:15.500 INFO:tasks.workunit.client.0.smithi031.stderr:listed 2 entries 2023-10-03T17:36:15.511 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:36:15.511 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:36:25.519 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:25.519 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:36:25.534 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:25.541 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:26.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:36:26.704 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:36:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354586 2023-10-03T17:36:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354586 -le 1696358112 ]] 2023-10-03T17:36:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:36:26.705 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:36:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:36.713 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:36.723 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:36:36.732 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:37.815 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:36:37.817 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:36:37.818 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354597 2023-10-03T17:36:37.819 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354597 -le 1696358112 ]] 2023-10-03T17:36:37.819 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:36:37.819 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:36:47.822 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:36:47.828 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:47.828 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:47.828 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354609 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354609 -le 1696358112 ]] 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:36:49.106 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:36:50.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.842+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:36:50.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.865+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:36:50.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.877+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:36:50.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.877+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:36:50.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.877+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:36:50.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.888+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:36:50.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.888+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:36:50.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.888+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:36:50.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.888+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:36:50.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:36:50.888+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:36:59.123 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:36:59.123 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:36:59.124 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:36:59.124 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:00.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.089+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcb500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcbb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.094 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcb700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bbf6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcbf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bbf6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcb900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.095 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.090+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.101 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.097+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.101 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.097+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcb500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.103 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.103 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724ba7a880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.103 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b1f1200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.103+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcb700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.106+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.106+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.109 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.109+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.109 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.109+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcbf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.112 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.112+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.112 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.112+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.128 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.121+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.128 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.121+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.128+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.128+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.128+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcb980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:37:00.128+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbf6700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:37:00.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:37:00.266 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:37:00.267 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354620 2023-10-03T17:37:00.267 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354620 -le 1696358112 ]] 2023-10-03T17:37:00.267 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:37:00.268 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:37:03.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.476+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:37:03.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.481+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:37:03.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.481+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:37:03.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.481+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:37:03.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.481+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:37:03.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:37:03.481+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:37:10.272 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:10.273 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:37:10.277 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:10.283 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:11.443 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2110891374 2023-10-03T17:37:11.444 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:37:11.444 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354631 2023-10-03T17:37:11.445 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354631 -le 1696358112 ]] 2023-10-03T17:37:11.445 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2110891374 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:37:11.445 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:37:21.452 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:37:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:21.453 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:22.805 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:37:22.808 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:37:22.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354642 2023-10-03T17:37:22.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354642 -le 1696358112 ]] 2023-10-03T17:37:22.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3463023637 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\1\1\0\8\9\1\3\7\4 ]] 2023-10-03T17:37:22.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3463023637 2023-10-03T17:37:25.573 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3463023637 until 2023-10-03T18:37:24.858107+0000 (3600 sec) 2023-10-03T17:37:25.604 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:37:25.605 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3463023637 2023-10-03T17:37:26.662 INFO:tasks.workunit.client.0.smithi031.stderr:listed 3 entries 2023-10-03T17:37:26.683 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:37:26.683 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:37:36.685 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:37:36.686 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:36.687 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:36.692 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:38.048 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:37:38.049 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:37:38.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354658 2023-10-03T17:37:38.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354658 -le 1696358112 ]] 2023-10-03T17:37:38.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3463023637 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\4\6\3\0\2\3\6\3\7 ]] 2023-10-03T17:37:38.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:37:48.057 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:37:48.074 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:48.074 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:37:48.074 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:37:49.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:37:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354669 2023-10-03T17:37:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354669 -le 1696358112 ]] 2023-10-03T17:37:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3463023637 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\4\6\3\0\2\3\6\3\7 ]] 2023-10-03T17:37:49.286 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:37:59.288 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:37:59.289 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:37:59.289 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:37:59.310 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.027+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.027+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724bbc4180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724afcb880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcb780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.049+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b946380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b946500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afcbd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b993a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.119+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bbc4080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.125 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.126 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.126 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724bbc4200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.126 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b993f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.126 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.120+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724bbc4a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.158 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.158+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.158 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:38:00.158+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724bbc4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:38:00.518 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:38:00.518 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:38:00.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354680 2023-10-03T17:38:00.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354680 -le 1696358112 ]] 2023-10-03T17:38:00.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3463023637 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\4\6\3\0\2\3\6\3\7 ]] 2023-10-03T17:38:00.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:38:10.525 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:10.526 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:38:10.548 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:10.548 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:11.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3463023637 2023-10-03T17:38:11.824 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:38:11.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354691 2023-10-03T17:38:11.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354691 -le 1696358112 ]] 2023-10-03T17:38:11.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3463023637 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\4\6\3\0\2\3\6\3\7 ]] 2023-10-03T17:38:11.826 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:38:20.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.105+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:38:20.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.113+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:38:20.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.113+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:38:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:38:20.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:20.131+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:38:21.946 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:38:21.946 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:21.946 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:21.955 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:22.543 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.542+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:38:22.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.545+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:38:22.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.552+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:38:22.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.552+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:38:22.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.552+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:38:22.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.566+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:38:22.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:38:22.603+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:38:23.266 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:38:23.266 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:38:23.271 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354703 2023-10-03T17:38:23.271 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354703 -le 1696358112 ]] 2023-10-03T17:38:23.271 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\4\6\3\0\2\3\6\3\7 ]] 2023-10-03T17:38:23.271 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/87597209 2023-10-03T17:38:25.578 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/87597209 until 2023-10-03T18:38:24.817103+0000 (3600 sec) 2023-10-03T17:38:25.600 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:38:25.612 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/87597209 2023-10-03T17:38:26.535 INFO:tasks.workunit.client.0.smithi031.stderr:listed 4 entries 2023-10-03T17:38:26.552 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:38:26.552 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:38:36.559 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:36.559 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:36.560 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:38:36.564 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:38.300 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:38:38.300 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:38:38.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354718 2023-10-03T17:38:38.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354718 -le 1696358112 ]] 2023-10-03T17:38:38.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:38:38.302 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:38:43.734 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:38:43.733+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:38:48.316 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:48.324 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:48.325 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:38:48.337 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:38:49.517 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:38:49.517 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:38:49.518 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354729 2023-10-03T17:38:49.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354729 -le 1696358112 ]] 2023-10-03T17:38:49.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:38:49.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:38:59.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:38:59.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:38:59.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:38:59.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:00.014 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.014+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.014 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.014+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.015 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.016+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.015 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.016+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.018+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.019+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.047 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.047+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.047+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.047+0000 7f57f88ef700 -1 librbd::ImageState: 0x557247bc0180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.047+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afcb580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.077 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.075+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.077 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.075+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b905f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.079+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724ba7a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.082+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b904d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x557247bc0400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x557247bc0600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x557247bc0880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b905f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724ba7a600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.120 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.120 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.120 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.121 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.121 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.121 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:39:00.120+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:39:00.741 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:39:00.741 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:39:00.742 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354740 2023-10-03T17:39:00.742 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354740 -le 1696358112 ]] 2023-10-03T17:39:00.742 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:39:00.742 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:39:10.756 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:39:10.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:10.795 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:10.822 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:11.986 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:39:11.987 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:39:11.987 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354751 2023-10-03T17:39:11.987 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354751 -le 1696358112 ]] 2023-10-03T17:39:11.988 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:39:11.988 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:39:18.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.774+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.839+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:39:18.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:39:18.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:39:18.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:39:18.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:39:18.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:18.841+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:39:21.995 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:21.996 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:22.016 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:22.044 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:39:23.211 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/87597209 2023-10-03T17:39:23.212 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:39:23.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354763 2023-10-03T17:39:23.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354763 -le 1696358112 ]] 2023-10-03T17:39:23.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/87597209 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:39:23.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:39:33.226 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:39:33.227 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:33.239 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:33.267 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:34.470 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:39:34.512 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:39:34.512 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354774 2023-10-03T17:39:34.513 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354774 -le 1696358112 ]] 2023-10-03T17:39:34.513 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\5\9\7\2\0\9 ]] 2023-10-03T17:39:34.513 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3781251104 2023-10-03T17:39:36.701 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3781251104 until 2023-10-03T18:39:35.985132+0000 (3600 sec) 2023-10-03T17:39:36.717 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:39:36.719 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3781251104 2023-10-03T17:39:37.721 INFO:tasks.workunit.client.0.smithi031.stderr:listed 5 entries 2023-10-03T17:39:37.734 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:39:37.735 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:39:47.750 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:39:47.751 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:39:47.755 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:47.775 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:49.033 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:39:49.035 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:39:49.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354789 2023-10-03T17:39:49.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354789 -le 1696358112 ]] 2023-10-03T17:39:49.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:39:49.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:39:51.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.296+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:39:51.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.303+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:39:51.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.303+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:39:51.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.313+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:39:51.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.315+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:39:51.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:39:51.315+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:39:59.046 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:39:59.046 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:39:59.048 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:39:59.056 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:00.016 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.015+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.016 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.016+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.016 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.016+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.016 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.016+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b947c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.016 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.016+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be46200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.017 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.016+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724ba7a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.023 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.019+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.023 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.019+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b947b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.030 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.030+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.030 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.030+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.030 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.030+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.031 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.030+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.031 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.031+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.031 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.031+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.059+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.059+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be46a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.080+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be46f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be46d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be47380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724be46800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.104 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.105 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.105 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.105 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.105 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.105 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.104+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.128 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.127+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.127+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.127+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.129 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:40:00.127+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be46d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:40:00.239 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:40:00.240 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:00.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354800 2023-10-03T17:40:00.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354800 -le 1696358112 ]] 2023-10-03T17:40:00.242 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:40:00.243 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:40:09.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.154+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:40:09.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.154+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:40:09.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.154+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:40:09.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.154+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:40:09.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.179+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:40:09.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.179+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:40:09.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.179+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:40:09.215 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.213+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:40:09.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:09.221+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:40:10.269 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:10.269 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:10.270 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:10.270 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:40:10.698 INFO:tasks.workunit.client.0.smithi031.stderr:2023-10-03T17:40:10.693+0000 7f68f259c700 0 --2- 172.21.15.31:0/4245889752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68dc007e10 0x7f68f4122300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-03T17:40:11.640 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:40:11.641 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:11.642 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354811 2023-10-03T17:40:11.642 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354811 -le 1696358112 ]] 2023-10-03T17:40:11.642 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:40:11.642 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:40:21.656 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:40:21.662 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:21.682 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:21.691 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:22.942 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:40:22.951 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:22.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354822 2023-10-03T17:40:22.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354822 -le 1696358112 ]] 2023-10-03T17:40:22.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:40:22.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:40:32.960 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:40:32.975 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:34.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3781251104 2023-10-03T17:40:34.213 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354834 2023-10-03T17:40:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354834 -le 1696358112 ]] 2023-10-03T17:40:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3781251104 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:40:34.214 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:40:39.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.617+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:40:39.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:40:39.629+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:40:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:40:44.219 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:44.223 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:45.484 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:40:45.484 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:45.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354845 2023-10-03T17:40:45.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354845 -le 1696358112 ]] 2023-10-03T17:40:45.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\7\8\1\2\5\1\1\0\4 ]] 2023-10-03T17:40:45.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/1609851556 2023-10-03T17:40:47.683 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/1609851556 until 2023-10-03T18:40:46.926751+0000 (3600 sec) 2023-10-03T17:40:47.689 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:40:47.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/1609851556 2023-10-03T17:40:48.633 INFO:tasks.workunit.client.0.smithi031.stderr:listed 6 entries 2023-10-03T17:40:48.645 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:40:48.645 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:40:52.779 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:40:52.779+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:40:52.779 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:40:52.779+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:40:52.779 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:40:52.779+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T17:40:58.648 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:40:58.648 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:40:58.664 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:40:58.666 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:40:59.814 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:40:59.815 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:40:59.815 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354859 2023-10-03T17:40:59.815 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354859 -le 1696358112 ]] 2023-10-03T17:40:59.815 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:40:59.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:00.007 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.007+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.007 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.008+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b1f1200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.019 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.019+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.019 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.019+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.020 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.019+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b947000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.020 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.019+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be46e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.021+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.021+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be47e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.028+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.028+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.028+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be47d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.028+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be98b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be98f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.045+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.046+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.065+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.065+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.065+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.065+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be98d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.072+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.072+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.072+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.073+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.073+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.073+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.073+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.073+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724be99e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.080+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.080+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c08a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.082+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.083+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.083+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c08a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.083+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c08a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.095+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:41:00.095+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c08a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:41:09.825 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:41:09.825 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:41:09.834 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:41:09.834 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:41:11.070 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:41:11.076 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:41:11.078 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354871 2023-10-03T17:41:11.078 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354871 -le 1696358112 ]] 2023-10-03T17:41:11.078 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:41:11.078 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:21.082 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:41:21.093 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:41:21.093 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:41:21.093 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:41:22.288 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:41:22.289 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:41:22.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354882 2023-10-03T17:41:22.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354882 -le 1696358112 ]] 2023-10-03T17:41:22.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:41:22.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:32.304 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:41:32.308 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:41:32.317 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:41:32.324 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:41:33.454 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:41:33.454 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:41:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354893 2023-10-03T17:41:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354893 -le 1696358112 ]] 2023-10-03T17:41:33.455 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:41:33.456 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:38.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.644+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:41:38.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.674+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:41:38.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.674+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:41:38.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.685+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:41:38.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.685+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:41:38.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:38.724+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:41:43.467 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:41:43.468 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:41:43.472 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:41:43.472 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:41:44.644 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1609851556 2023-10-03T17:41:44.673 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:41:44.684 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354904 2023-10-03T17:41:44.684 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354904 -le 1696358112 ]] 2023-10-03T17:41:44.684 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1609851556 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:41:44.685 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:54.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:54.471+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:41:54.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:54.471+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:41:54.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:54.471+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:41:54.472 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:41:54.471+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:41:54.686 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:41:54.695 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:41:54.707 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:41:54.707 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:41:55.868 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:41:55.870 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:41:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354915 2023-10-03T17:41:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354915 -le 1696358112 ]] 2023-10-03T17:41:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\0\9\8\5\1\5\5\6 ]] 2023-10-03T17:41:55.872 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/1130397553 2023-10-03T17:41:58.704 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/1130397553 until 2023-10-03T18:41:57.770063+0000 (3600 sec) 2023-10-03T17:41:58.712 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:41:58.714 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/1130397553 2023-10-03T17:41:59.727 INFO:tasks.workunit.client.0.smithi031.stderr:listed 7 entries 2023-10-03T17:41:59.764 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:41:59.764 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:41:59.766 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:41:59.765+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:41:59.766 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:41:59.765+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:42:00.065 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.061+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.061+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.061+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.061+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be98980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b5e4580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b904a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.079+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.084+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.084+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.084+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.085+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.085+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724be98980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.085+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.085+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.086+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b5e4580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.089+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.089+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.113 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.113+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.113 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.113+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.113 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.113+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b904a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:00.113 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:42:00.113+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724bb1e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:42:09.767 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:42:09.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:42:09.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:42:09.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:42:10.945 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:42:10.945 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:42:10.947 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354930 2023-10-03T17:42:10.947 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354930 -le 1696358112 ]] 2023-10-03T17:42:10.947 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:42:10.947 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:42:20.958 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:42:20.960 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:42:20.966 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:42:20.967 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:42:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:42:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:42:22.027 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354942 2023-10-03T17:42:22.028 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354942 -le 1696358112 ]] 2023-10-03T17:42:22.028 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:42:22.028 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:42:32.038 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:42:32.039 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:42:32.040 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:42:32.058 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:42:32.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:32.717+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:42:32.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:32.717+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:42:32.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:32.718+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:42:32.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:32.721+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:42:32.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:32.734+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:42:33.276 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:42:33.277 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:42:33.277 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354953 2023-10-03T17:42:33.277 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354953 -le 1696358112 ]] 2023-10-03T17:42:33.277 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:42:33.278 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:42:43.279 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:42:43.280 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:42:43.298 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:42:43.301 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:42:44.475 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:42:44.476 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:42:44.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354964 2023-10-03T17:42:44.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354964 -le 1696358112 ]] 2023-10-03T17:42:44.485 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:42:44.486 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:42:54.493 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:42:54.493 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:42:54.493 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:42:54.498 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:42:55.796 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:42:55.796 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:42:55.798 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354975 2023-10-03T17:42:55.798 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354975 -le 1696358112 ]] 2023-10-03T17:42:55.798 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:42:55.798 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:42:57.742 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.737+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:42:57.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.750+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:42:57.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.750+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:42:57.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.750+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:42:57.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.753+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:42:57.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:42:57.773+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:43:05.800 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:43:05.801 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:43:05.801 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:43:05.803 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:43:06.832 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:06.832+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:43:06.845 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:06.845+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:43:06.861 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:06.861+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1130397553 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354987 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354987 -le 1696358112 ]] 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1130397553 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:43:07.050 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:43:14.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.058+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:43:14.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.059+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:43:14.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:43:14.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:14.065+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:43:17.069 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:43:17.069 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:43:17.075 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:43:17.075 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:43:18.305 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:43:18.305 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:43:18.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696354998 2023-10-03T17:43:18.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696354998 -le 1696358112 ]] 2023-10-03T17:43:18.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\1\3\0\3\9\7\5\5\3 ]] 2023-10-03T17:43:18.306 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/876723378 2023-10-03T17:43:18.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.836+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:43:18.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.846+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:43:18.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.846+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:43:18.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.846+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:43:18.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.846+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:43:18.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.862+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:43:18.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:43:18.862+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:43:20.725 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/876723378 until 2023-10-03T18:43:20.007132+0000 (3600 sec) 2023-10-03T17:43:20.752 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:43:20.758 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/876723378 2023-10-03T17:43:21.779 INFO:tasks.workunit.client.0.smithi031.stderr:listed 8 entries 2023-10-03T17:43:21.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:43:21.797 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:43:31.804 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:43:31.805 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:43:31.805 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:43:31.821 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:43:32.960 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:43:32.960 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:43:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355012 2023-10-03T17:43:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355012 -le 1696358112 ]] 2023-10-03T17:43:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:43:32.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:43:42.969 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:43:42.971 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:43:42.980 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:43:42.987 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:43:44.278 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:43:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:43:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355024 2023-10-03T17:43:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355024 -le 1696358112 ]] 2023-10-03T17:43:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:43:44.279 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:43:48.846 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:48.841+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:43:48.846 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:48.841+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:43:48.864 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:43:48.864+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:43:54.281 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:43:54.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:43:54.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:43:54.293 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:43:55.437 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:43:55.440 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:43:55.440 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355035 2023-10-03T17:43:55.440 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355035 -le 1696358112 ]] 2023-10-03T17:43:55.440 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:43:55.441 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:43:56.196 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:43:56.196+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:43:56.196 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:43:56.196+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:44:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.035+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.035+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724af0c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.050+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.050+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.050+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.050+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b947880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b946180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b946e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c026180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724af0c000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.061 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.062+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.062 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.062+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.062 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b947880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b946180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.063+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b946e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.081+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.081+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.081+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c026180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c027b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c027900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:00.096 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:44:00.096+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c027f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:44:05.448 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:44:05.448 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:44:05.448 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:44:05.456 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:44:06.696 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:44:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:44:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355046 2023-10-03T17:44:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355046 -le 1696358112 ]] 2023-10-03T17:44:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:44:06.720 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:44:07.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.714+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:44:07.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.714+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:44:07.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.741+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:44:07.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.741+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:44:07.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.742+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:44:07.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.745+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:44:07.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.745+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:44:07.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.761+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:44:07.766 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.767+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:44:07.776 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:07.772+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:16.739 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:44:16.740 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:44:16.740 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:44:16.751 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:44:17.956 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/876723378 2023-10-03T17:44:17.958 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:44:17.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355057 2023-10-03T17:44:17.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355057 -le 1696358112 ]] 2023-10-03T17:44:17.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/876723378 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:44:17.962 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:44:19.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.659+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:44:19.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.705+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:44:19.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.705+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:44:19.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.735+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:44:19.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.735+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:44:19.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:19.741+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:27.966 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:44:27.973 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:44:27.983 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:44:27.983 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:44:28.106 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:44:28.093+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T17:44:28.106 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:44:28.093+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T17:44:28.106 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:44:28.093+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:44:28.106 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:44:28.093+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:44:28.106 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:44:28.093+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:44:29.120 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:44:29.122 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:44:29.122 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355069 2023-10-03T17:44:29.122 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355069 -le 1696358112 ]] 2023-10-03T17:44:29.122 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2304558628 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\6\7\2\3\3\7\8 ]] 2023-10-03T17:44:29.122 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2304558628 2023-10-03T17:44:31.800 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2304558628 until 2023-10-03T18:44:31.042243+0000 (3600 sec) 2023-10-03T17:44:31.828 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:44:31.828 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2304558628 2023-10-03T17:44:32.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:32.372+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:44:32.461 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:32.383+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:32.800 INFO:tasks.workunit.client.0.smithi031.stderr:listed 9 entries 2023-10-03T17:44:32.857 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:44:32.857 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:44:35.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.349+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:44:35.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:35.366+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:40.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:40.484+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:42.823 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:44:42.823 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:44:42.828 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:44:42.838 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:44:44.032 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:44:44.034 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:44:44.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355084 2023-10-03T17:44:44.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355084 -le 1696358112 ]] 2023-10-03T17:44:44.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2304558628 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\3\0\4\5\5\8\6\2\8 ]] 2023-10-03T17:44:44.035 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:44:47.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.713+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:44:47.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.713+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:44:47.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.737+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:44:47.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.737+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:44:47.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.737+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:44:47.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.747+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:44:47.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:44:47.747+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:44:54.035 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:44:54.039 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:44:54.062 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:44:54.071 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:44:55.318 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:44:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:44:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355095 2023-10-03T17:44:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355095 -le 1696358112 ]] 2023-10-03T17:44:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2304558628 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\3\0\4\5\5\8\6\2\8 ]] 2023-10-03T17:44:55.319 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:45:00.038 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c292a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c292c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c292e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.044+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.062 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.063+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.063+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.063+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.063+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c292a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c292c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c292e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293400 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:00.083+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c293c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:05.322 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:45:05.322 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:45:05.323 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:45:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:45:06.516 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:45:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:45:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355106 2023-10-03T17:45:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355106 -le 1696358112 ]] 2023-10-03T17:45:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2304558628 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\3\0\4\5\5\8\6\2\8 ]] 2023-10-03T17:45:06.517 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:45:16.526 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:45:16.526 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:45:16.526 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:45:16.526 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:45:17.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.601+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:45:17.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.665+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:45:17.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.665+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:45:17.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.671+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:45:17.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.705+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:45:17.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.716+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:45:17.763 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2304558628 2023-10-03T17:45:17.771 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:45:17.772 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355117 2023-10-03T17:45:17.772 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355117 -le 1696358112 ]] 2023-10-03T17:45:17.772 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2304558628 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\3\0\4\5\5\8\6\2\8 ]] 2023-10-03T17:45:17.772 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:45:17.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.775+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:45:17.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.775+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:45:17.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.775+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:45:17.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.775+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:45:17.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:17.775+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:45:27.777 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:45:27.777 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:45:27.777 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:45:27.777 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:45:28.909 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2909175 2023-10-03T17:45:28.910 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:45:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355128 2023-10-03T17:45:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355128 -le 1696358112 ]] 2023-10-03T17:45:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2909175 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\3\0\4\5\5\8\6\2\8 ]] 2023-10-03T17:45:28.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2909175 2023-10-03T17:45:29.697 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:29.697+0000 7f57f88ef700 -1 librbd::mirror::GetInfoRequest: 0x55724c1cf200 handle_get_snapcontext: failed to get snapcontext: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:29.703 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:29.703+0000 7f57f80ee700 -1 librbd::mirror::GetInfoRequest: 0x55724c1cf2c0 handle_get_mirror_image: failed to retrieve mirroring state: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:29.703 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:45:29.703+0000 7f57f58e9700 -1 librbd::api::Mirror: image_info_list: failed to list mirror image statuses: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:45:30.713 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2909175 until 2023-10-03T18:45:29.988251+0000 (3600 sec) 2023-10-03T17:45:30.732 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:45:30.759 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2909175 2023-10-03T17:45:31.672 INFO:tasks.workunit.client.0.smithi031.stderr:listed 10 entries 2023-10-03T17:45:31.689 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2909175 2023-10-03T17:45:31.689 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:45:39.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:39.004+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:45:39.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:39.004+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:45:40.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:40.726+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:45:40.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:40.726+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:45:40.742 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:40.741+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:45:40.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:40.757+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:45:41.695 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:45:41.695 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:45:41.696 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:45:41.703 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:45:42.903 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:45:42.903 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:45:42.904 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355142 2023-10-03T17:45:42.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355142 -le 1696358112 ]] 2023-10-03T17:45:42.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\9\0\9\1\7\5 ]] 2023-10-03T17:45:42.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/939425469 2023-10-03T17:45:44.871 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/939425469 until 2023-10-03T18:45:44.118199+0000 (3600 sec) 2023-10-03T17:45:44.899 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:45:44.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/939425469 2023-10-03T17:45:45.892 INFO:tasks.workunit.client.0.smithi031.stderr:listed 11 entries 2023-10-03T17:45:45.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:45:45.911 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:45:54.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:45:54.341+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:45:55.914 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:45:55.914 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:45:55.914 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:45:55.916 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:45:57.081 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:45:57.081 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:45:57.082 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355157 2023-10-03T17:45:57.082 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355157 -le 1696358112 ]] 2023-10-03T17:45:57.082 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:45:57.082 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:46:00.010 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.010+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.010 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.011+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c293780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.055 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.055 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.055 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.055 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.055 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.056 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.056 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.056 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.056 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.056 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e8d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e9880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.060+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e9a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e9c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e9e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.072+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c23a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.074+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.074+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.074+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.074+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23a480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.075+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:46:00.075+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:46:02.135 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:46:02.134+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:46:02.170 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:46:02.171+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:46:02.198 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:46:02.197+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:46:07.085 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:46:07.085 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:46:07.094 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:46:07.094 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:46:08.308 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:46:08.310 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:46:08.311 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355168 2023-10-03T17:46:08.312 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355168 -le 1696358112 ]] 2023-10-03T17:46:08.312 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:46:08.312 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:46:18.324 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:46:18.325 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:46:18.329 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:46:18.361 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:46:19.583 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:46:19.587 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:46:19.589 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355179 2023-10-03T17:46:19.589 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355179 -le 1696358112 ]] 2023-10-03T17:46:19.589 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:46:19.590 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:46:29.591 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:46:29.598 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:46:29.598 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:46:29.598 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:46:30.699 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:46:30.702 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:46:30.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355190 2023-10-03T17:46:30.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355190 -le 1696358112 ]] 2023-10-03T17:46:30.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:46:30.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:46:33.967 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:33.960+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:46:33.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:33.960+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:46:33.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:33.971+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:46:34.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.030+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:46:34.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.030+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:46:34.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.030+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:46:34.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.030+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:46:34.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.032+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:46:34.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:46:34.032+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:46:37.033 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:46:37.032+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:46:37.033 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:46:37.032+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:46:37.033 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:46:37.032+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T17:46:37.033 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:46:37.032+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T17:46:40.706 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:46:40.707 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:46:40.714 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:46:40.735 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:46:41.942 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/939425469 2023-10-03T17:46:41.943 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:46:41.943 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355201 2023-10-03T17:46:41.944 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355201 -le 1696358112 ]] 2023-10-03T17:46:41.944 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/939425469 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:46:41.944 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:46:51.949 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:46:51.958 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:46:51.968 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:46:51.968 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:46:53.224 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:46:53.224 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:46:53.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355213 2023-10-03T17:46:53.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355213 -le 1696358112 ]] 2023-10-03T17:46:53.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3594551136 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\3\9\4\2\5\4\6\9 ]] 2023-10-03T17:46:53.225 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3594551136 2023-10-03T17:46:55.991 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3594551136 until 2023-10-03T18:46:55.213797+0000 (3600 sec) 2023-10-03T17:46:56.002 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:46:56.006 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3594551136 2023-10-03T17:46:56.921 INFO:tasks.workunit.client.0.smithi031.stderr:listed 12 entries 2023-10-03T17:46:56.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:46:56.951 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:00.007 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.008+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.008 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.008+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c23bd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.024+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.024+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c23bf00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.024+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.025+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.025+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c276100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.025+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.032 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.032+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.032 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.032+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.033+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.034+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.034+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.071 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.069+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.069+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.069+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.070+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c276d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.070+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c276f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.070+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.071+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.071+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.071+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c277680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.071+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c277480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.083+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.083+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.083+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.083+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.088+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.088+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.088+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.089+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.089+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.090 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.089+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277f00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.092 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.091+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.092 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.091+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.092 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.091+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c326100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:00.092 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:47:00.091+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c326300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:47:06.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:47:06.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:47:06.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:47:06.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:47:06.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:47:06.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:06.228+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:47:06.962 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:47:06.965 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:47:06.965 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:47:06.977 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:47:08.141 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:47:08.142 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:47:08.143 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355228 2023-10-03T17:47:08.143 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355228 -le 1696358112 ]] 2023-10-03T17:47:08.143 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3594551136 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\5\9\4\5\5\1\1\3\6 ]] 2023-10-03T17:47:08.143 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:09.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.563+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:47:09.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.567+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:47:09.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.588+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:47:09.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.650+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:47:09.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.655+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:47:09.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.655+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:47:09.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.667+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:47:09.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.689+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:47:09.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:09.715+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:47:14.734 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:47:14.734+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:47:14.734 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:47:14.734+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:47:14.734 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:47:14.734+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T17:47:18.148 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:47:18.148 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:47:18.157 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:47:18.158 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:47:19.401 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:47:19.402 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:47:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355239 2023-10-03T17:47:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355239 -le 1696358112 ]] 2023-10-03T17:47:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3594551136 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\5\9\4\5\5\1\1\3\6 ]] 2023-10-03T17:47:19.404 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:29.414 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:47:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:47:29.416 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:47:29.419 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:47:30.553 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:47:30.553 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:47:30.554 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355250 2023-10-03T17:47:30.554 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355250 -le 1696358112 ]] 2023-10-03T17:47:30.554 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3594551136 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\5\9\4\5\5\1\1\3\6 ]] 2023-10-03T17:47:30.554 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:33.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.066+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:47:33.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.073+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:47:33.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.073+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:47:33.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.073+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:47:33.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.166+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:47:33.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.166+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:47:33.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.169+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:47:33.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.169+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:47:33.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:33.172+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:47:35.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.714+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:47:35.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.714+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:47:35.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.714+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:47:35.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.726+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:47:35.733 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.726+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:47:35.733 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.727+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:47:35.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.736+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:47:35.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:35.736+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:47:40.559 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:47:40.560 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:47:40.560 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:47:40.560 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:47:41.902 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3594551136 2023-10-03T17:47:41.903 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:47:41.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355261 2023-10-03T17:47:41.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355261 -le 1696358112 ]] 2023-10-03T17:47:41.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3594551136 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\5\9\4\5\5\1\1\3\6 ]] 2023-10-03T17:47:41.905 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:47.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.041+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:47:47.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.061+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:47:47.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.061+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:47:47.079 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.079+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:47:47.086 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.085+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:47:47.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.094+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:47:47.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.094+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:47:47.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.094+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:47:47.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.094+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:47:47.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:47.094+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:47:51.907 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:47:51.910 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:47:51.910 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:47:51.921 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:47:53.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:47:53.134 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:47:53.134 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355273 2023-10-03T17:47:53.134 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355273 -le 1696358112 ]] 2023-10-03T17:47:53.134 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\5\9\4\5\5\1\1\3\6 ]] 2023-10-03T17:47:53.134 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/166292732 2023-10-03T17:47:54.963 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/166292732 until 2023-10-03T18:47:54.246471+0000 (3600 sec) 2023-10-03T17:47:54.979 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:47:54.979 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/166292732 2023-10-03T17:47:55.953 INFO:tasks.workunit.client.0.smithi031.stderr:listed 13 entries 2023-10-03T17:47:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:47:55.970 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:47:57.645 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:47:57.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T17:47:57.646 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:47:57.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:47:57.646 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:47:57.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:47:57.646 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:47:57.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:47:58.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:47:58.953+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:48:00.023 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.023 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.024 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.020+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79f880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.034+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.041 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.034+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79f380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.046+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79f700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.066+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c08aa80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c027880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b79ff80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b79fd80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b993c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b946500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b5e4e80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.068+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.069+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.069+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.071+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:48:00.071+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:48:05.972 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:48:05.973 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:48:05.981 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:48:05.981 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:48:07.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:48:07.181 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:48:07.192 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355287 2023-10-03T17:48:07.192 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355287 -le 1696358112 ]] 2023-10-03T17:48:07.192 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:48:07.192 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:48:17.199 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:48:17.200 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:48:17.200 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:48:17.200 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:48:18.341 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:48:18.342 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:48:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355298 2023-10-03T17:48:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355298 -le 1696358112 ]] 2023-10-03T17:48:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:48:18.343 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:48:26.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.399+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:48:26.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.399+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:48:26.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.399+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:48:26.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.399+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:48:26.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.406+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:48:26.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.406+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:48:26.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.406+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:48:26.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:26.406+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:48:28.351 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:48:28.351 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:48:28.351 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:48:28.362 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:48:29.534 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:48:29.534 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:48:29.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355309 2023-10-03T17:48:29.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355309 -le 1696358112 ]] 2023-10-03T17:48:29.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:48:29.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:48:36.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:48:36.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:48:36.361+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:48:39.546 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:48:39.546 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:48:39.546 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:48:39.550 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:48:40.781 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:48:40.782 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:48:40.784 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355320 2023-10-03T17:48:40.784 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355320 -le 1696358112 ]] 2023-10-03T17:48:40.785 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:48:40.785 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:48:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:48:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:48:50.798 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:48:50.801 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:48:51.998 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:48:51.998 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:48:51.999 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355331 2023-10-03T17:48:51.999 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355331 -le 1696358112 ]] 2023-10-03T17:48:51.999 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:48:51.999 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:02.002 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:49:02.011 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:49:02.011 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:49:02.014 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:49:03.241 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/166292732 2023-10-03T17:49:03.242 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:49:03.244 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355343 2023-10-03T17:49:03.244 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355343 -le 1696358112 ]] 2023-10-03T17:49:03.244 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/166292732 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:49:03.244 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:13.245 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:49:13.246 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:49:13.246 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:49:13.246 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:49:14.469 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:49:14.469 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:49:14.469 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355354 2023-10-03T17:49:14.469 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355354 -le 1696358112 ]] 2023-10-03T17:49:14.470 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3379124484 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\6\6\2\9\2\7\3\2 ]] 2023-10-03T17:49:14.470 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3379124484 2023-10-03T17:49:17.080 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3379124484 until 2023-10-03T18:49:16.321829+0000 (3600 sec) 2023-10-03T17:49:17.096 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:49:17.097 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3379124484 2023-10-03T17:49:18.024 INFO:tasks.workunit.client.0.smithi031.stderr:listed 14 entries 2023-10-03T17:49:18.057 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:49:18.058 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:28.060 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:49:28.061 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:49:28.085 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:49:28.091 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:49:29.244 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:49:29.246 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:49:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355369 2023-10-03T17:49:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355369 -le 1696358112 ]] 2023-10-03T17:49:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3379124484 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\3\7\9\1\2\4\4\8\4 ]] 2023-10-03T17:49:29.247 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:35.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:35.519+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:49:35.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:35.519+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:49:35.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:35.520+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:49:35.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:35.520+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:49:35.533 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:35.531+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:49:39.249 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:49:39.249 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:49:39.253 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:49:39.263 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:49:40.449 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:49:40.449 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:49:40.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355380 2023-10-03T17:49:40.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355380 -le 1696358112 ]] 2023-10-03T17:49:40.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3379124484 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\3\7\9\1\2\4\4\8\4 ]] 2023-10-03T17:49:40.450 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:47.019 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:49:47.014+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T17:49:47.019 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:49:47.014+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T17:49:47.019 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:49:47.014+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:49:47.019 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:49:47.014+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:49:47.019 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:49:47.014+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:49:50.456 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:49:50.463 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:49:50.463 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:49:50.481 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:49:51.702 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:49:51.702 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:49:51.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355391 2023-10-03T17:49:51.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355391 -le 1696358112 ]] 2023-10-03T17:49:51.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3379124484 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\3\7\9\1\2\4\4\8\4 ]] 2023-10-03T17:49:51.703 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:49:58.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.630+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:49:58.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.639+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:49:58.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.639+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:49:58.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.639+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:49:58.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.655+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:49:58.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:49:58.670+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:50:00.013 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.007+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.013 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.007+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.014 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.013+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.014 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.013+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.028 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.040 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.072 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.074 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12eb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.075 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12ed80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.071+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.080+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.080+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12fd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.085 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12ea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.086 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.087 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.084+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.135 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.133+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.135 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.133+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.135 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.133+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:00.135 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:50:00.133+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c12f500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:50:01.705 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:50:01.706 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:50:01.706 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:50:01.706 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:50:02.860 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3379124484 2023-10-03T17:50:02.860 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:50:02.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355402 2023-10-03T17:50:02.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355402 -le 1696358112 ]] 2023-10-03T17:50:02.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3379124484 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\3\7\9\1\2\4\4\8\4 ]] 2023-10-03T17:50:02.861 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:50:12.866 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:50:12.872 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:50:12.872 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:50:12.878 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:50:14.165 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:50:14.165 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:50:14.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355414 2023-10-03T17:50:14.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355414 -le 1696358112 ]] 2023-10-03T17:50:14.166 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\3\7\9\1\2\4\4\8\4 ]] 2023-10-03T17:50:14.167 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/1378966707 2023-10-03T17:50:16.093 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/1378966707 until 2023-10-03T18:50:15.347628+0000 (3600 sec) 2023-10-03T17:50:16.100 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:50:16.101 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/1378966707 2023-10-03T17:50:17.109 INFO:tasks.workunit.client.0.smithi031.stderr:listed 15 entries 2023-10-03T17:50:17.126 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:50:17.126 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:50:27.169 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:50:27.172 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:50:27.187 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:50:27.187 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:50:28.375 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:50:28.375 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:50:28.376 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355428 2023-10-03T17:50:28.376 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355428 -le 1696358112 ]] 2023-10-03T17:50:28.376 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:50:28.377 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:50:38.413 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:50:38.413 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:50:38.417 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:50:38.418 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:50:39.669 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:50:39.671 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:50:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355439 2023-10-03T17:50:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355439 -le 1696358112 ]] 2023-10-03T17:50:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:50:39.678 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:50:49.683 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:50:49.694 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:50:49.694 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:50:49.722 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:50:50.824 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:50:50.824 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:50:50.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355450 2023-10-03T17:50:50.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355450 -le 1696358112 ]] 2023-10-03T17:50:50.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:50:50.825 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:00.042 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.034+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.043 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.034+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c026800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.043 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.043 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.043 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.046 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.039+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.051+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.052 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.051+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c026800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12fe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12f180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.052+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.059+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.059+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.059+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.059+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:51:00.077+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:51:00.859 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:51:00.859 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:51:00.859 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:51:00.877 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:51:02.130 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:51:02.130 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:51:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355462 2023-10-03T17:51:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355462 -le 1696358112 ]] 2023-10-03T17:51:02.131 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:51:02.132 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:02.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.455+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:51:02.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.468+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:51:02.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.474+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:51:02.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.487+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:51:02.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.534+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:51:02.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:02.546+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:51:05.372 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:51:05.371+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:51:05.388 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:51:05.388+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:51:12.146 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:51:12.147 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:51:12.150 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:51:12.160 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:51:13.290 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1378966707 2023-10-03T17:51:13.290 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:51:13.292 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355473 2023-10-03T17:51:13.292 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355473 -le 1696358112 ]] 2023-10-03T17:51:13.292 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1378966707 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:51:13.292 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:23.298 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:51:23.298 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:51:23.305 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:51:23.306 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:51:24.418 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:51:24.418 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:51:24.419 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355484 2023-10-03T17:51:24.419 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355484 -le 1696358112 ]] 2023-10-03T17:51:24.419 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2507430896 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\3\7\8\9\6\6\7\0\7 ]] 2023-10-03T17:51:24.419 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2507430896 2023-10-03T17:51:27.087 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2507430896 until 2023-10-03T18:51:26.369132+0000 (3600 sec) 2023-10-03T17:51:27.102 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2507430896 2023-10-03T17:51:27.102 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:51:28.043 INFO:tasks.workunit.client.0.smithi031.stderr:listed 16 entries 2023-10-03T17:51:28.061 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:51:28.061 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:36.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:36.979+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:51:37.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.029+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:51:37.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.033+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:51:37.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.035+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:51:37.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:37.047+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:51:38.074 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:51:38.082 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:51:38.082 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:51:38.082 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:51:39.286 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:51:39.286 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:51:39.287 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355499 2023-10-03T17:51:39.288 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355499 -le 1696358112 ]] 2023-10-03T17:51:39.288 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2507430896 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\0\7\4\3\0\8\9\6 ]] 2023-10-03T17:51:39.288 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:51:49.292 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:51:49.297 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:51:49.308 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:51:50.503 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:51:50.505 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:51:50.505 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355510 2023-10-03T17:51:50.505 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355510 -le 1696358112 ]] 2023-10-03T17:51:50.505 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2507430896 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\0\7\4\3\0\8\9\6 ]] 2023-10-03T17:51:50.505 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:51:58.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.484+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:51:58.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.552+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:51:58.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.552+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:51:58.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:51:58.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:51:58.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:51:58.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:51:58.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:51:58.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:51:58.558+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:52:00.035 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.026+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.035 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.026+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.035 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.026+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.035 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.026+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b79e980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b993580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b79f600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c292600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c293980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.034+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724ad6ff00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.045+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.045+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.066 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b5e4380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e600 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:52:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c12e680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:52:00.533 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:00.534 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:00.540 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:00.548 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:52:01.760 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:52:01.760 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:52:01.761 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355521 2023-10-03T17:52:01.761 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355521 -le 1696358112 ]] 2023-10-03T17:52:01.761 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2507430896 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\0\7\4\3\0\8\9\6 ]] 2023-10-03T17:52:01.762 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:52:11.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:11.768 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:52:11.769 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:11.810 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:12.975 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2507430896 2023-10-03T17:52:12.981 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:52:12.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355532 2023-10-03T17:52:12.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355532 -le 1696358112 ]] 2023-10-03T17:52:12.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2507430896 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\0\7\4\3\0\8\9\6 ]] 2023-10-03T17:52:12.983 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:52:13.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:13.917+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:52:13.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:13.946+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:52:13.948 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:13.946+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:52:13.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:13.952+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:52:13.999 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:13.998+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:52:14.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:14.024+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:52:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:22.988 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:22.989 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:22.999 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:52:24.208 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:52:24.211 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:52:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355544 2023-10-03T17:52:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355544 -le 1696358112 ]] 2023-10-03T17:52:24.212 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\0\7\4\3\0\8\9\6 ]] 2023-10-03T17:52:24.213 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3173308107 2023-10-03T17:52:26.131 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3173308107 until 2023-10-03T18:52:25.427073+0000 (3600 sec) 2023-10-03T17:52:26.148 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3173308107 2023-10-03T17:52:26.156 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:52:27.040 INFO:tasks.workunit.client.0.smithi031.stderr:listed 17 entries 2023-10-03T17:52:27.057 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:52:27.057 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:52:30.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:52:30.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:52:30.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:52:30.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:52:30.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:30.900+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:52:37.070 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:37.070 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:37.082 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:52:37.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:38.231 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:52:38.231 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:52:38.232 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355558 2023-10-03T17:52:38.232 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355558 -le 1696358112 ]] 2023-10-03T17:52:38.232 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:52:38.233 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:52:47.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.768+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:52:47.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.770+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:52:47.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.770+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:52:47.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.777+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:52:47.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.800+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:52:47.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.800+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:52:47.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.824+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:52:47.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:52:47.840+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:52:48.234 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:48.240 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:48.240 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:48.240 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:52:49.413 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:52:49.414 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:52:49.414 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355569 2023-10-03T17:52:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355569 -le 1696358112 ]] 2023-10-03T17:52:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:52:49.415 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:52:59.418 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:52:59.418 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:52:59.418 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:52:59.424 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:00.004 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.004+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.004 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.004+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c027800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.020 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b905c00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b946a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.021 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.020+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b904880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.026+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.026+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.026+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.026+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c292480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.032 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.032+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.032 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.032+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c327000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.033 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.033+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.033 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.033+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c326c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e9480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c326800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c277d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08b280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c08a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bb1f200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79f680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.069 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.065+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b79ef80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.078+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b992200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.078 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:53:00.078+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c2e9d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:53:00.687 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:53:00.689 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:00.690 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355580 2023-10-03T17:53:00.690 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355580 -le 1696358112 ]] 2023-10-03T17:53:00.690 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:53:00.690 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:53:02.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:02.278+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:53:10.128 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:53:10.122+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T17:53:10.130 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:53:10.129+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:53:10.133 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:53:10.133+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:53:10.146 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:53:10.145+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:53:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:53:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:53:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:10.697 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:53:11.918 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:53:11.918 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:11.919 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355591 2023-10-03T17:53:11.919 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355591 -le 1696358112 ]] 2023-10-03T17:53:11.919 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:53:11.919 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:53:21.923 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:21.929 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:53:21.930 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:53:21.930 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:53:23.130 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3173308107 2023-10-03T17:53:23.137 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:23.138 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355603 2023-10-03T17:53:23.138 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355603 -le 1696358112 ]] 2023-10-03T17:53:23.138 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3173308107 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:53:23.138 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:53:30.273 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:53:30.271+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:53:30.276 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:53:30.276+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:53:33.139 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:53:33.140 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:53:33.150 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:53:33.160 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:34.332 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:53:34.340 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:34.341 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355614 2023-10-03T17:53:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355614 -le 1696358112 ]] 2023-10-03T17:53:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\7\3\3\0\8\1\0\7 ]] 2023-10-03T17:53:34.342 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/902319174 2023-10-03T17:53:36.182 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/902319174 until 2023-10-03T18:53:35.397148+0000 (3600 sec) 2023-10-03T17:53:36.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:53:36.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/902319174 2023-10-03T17:53:37.183 INFO:tasks.workunit.client.0.smithi031.stderr:listed 18 entries 2023-10-03T17:53:37.201 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:53:37.201 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:53:43.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.517+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:53:43.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.528+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:53:43.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.528+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:53:43.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.541+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:53:43.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.558+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:53:43.593 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:53:43.593+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:53:47.206 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:53:47.208 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:47.208 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:53:47.208 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:53:48.385 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:53:48.385 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:48.386 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355628 2023-10-03T17:53:48.386 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355628 -le 1696358112 ]] 2023-10-03T17:53:48.386 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:53:48.386 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:53:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:53:58.391 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:53:58.404 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:53:58.410 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:53:59.622 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:53:59.628 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:53:59.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355639 2023-10-03T17:53:59.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355639 -le 1696358112 ]] 2023-10-03T17:53:59.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:53:59.629 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:54:00.017 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.016+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.017 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.016+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.017 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.016+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c333580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.017 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.016+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.025 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.024+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.026 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.024+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c333880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.036 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c333b80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.037 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.036+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c333d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.050 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.049+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.050 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.049+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.050 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.049+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.050+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c333f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.050+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.051 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.050+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598c80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598d80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.057+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b598f80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.067 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b599180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b599380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.068 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.067+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b599580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.069+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.069+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b599780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.076+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:00.076 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:54:00.076+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b599980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:54:09.631 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:54:09.632 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:54:09.644 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:54:09.644 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:54:10.839 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:54:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:54:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355650 2023-10-03T17:54:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355650 -le 1696358112 ]] 2023-10-03T17:54:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:54:10.841 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:54:12.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.334+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:54:12.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.341+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:54:12.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.346+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:54:12.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.346+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:54:12.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.346+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:54:12.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.346+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:54:12.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:12.346+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:54:20.843 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:54:20.847 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:54:20.853 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:54:20.863 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:54:21.992 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:54:21.993 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:54:21.993 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355661 2023-10-03T17:54:21.993 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355661 -le 1696358112 ]] 2023-10-03T17:54:21.993 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:54:21.993 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:54:24.082 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:54:24.081+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:54:24.103 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:54:24.101+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:54:24.103 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:54:24.101+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:54:28.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.240+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:54:28.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.297+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:54:28.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.317+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.317+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.317+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.318+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.318+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.318+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.318+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:54:28.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:54:28.318+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:54:31.999 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:54:31.999 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:54:32.000 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:54:32.000 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:54:33.146 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/902319174 2023-10-03T17:54:33.146 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:54:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355673 2023-10-03T17:54:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355673 -le 1696358112 ]] 2023-10-03T17:54:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/902319174 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:54:33.152 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:54:43.162 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:54:43.173 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:54:43.173 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:54:43.174 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:54:44.327 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:54:44.328 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:54:44.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355684 2023-10-03T17:54:44.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355684 -le 1696358112 ]] 2023-10-03T17:54:44.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1789969421 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\9\0\2\3\1\9\1\7\4 ]] 2023-10-03T17:54:44.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/1789969421 2023-10-03T17:54:46.290 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/1789969421 until 2023-10-03T18:54:45.525121+0000 (3600 sec) 2023-10-03T17:54:46.307 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:54:46.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/1789969421 2023-10-03T17:54:47.279 INFO:tasks.workunit.client.0.smithi031.stderr:listed 19 entries 2023-10-03T17:54:47.295 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:54:47.295 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:54:48.593 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:54:48.593+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T17:54:48.593 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:54:48.593+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:54:48.593 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:54:48.593+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T17:54:48.593 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:54:48.593+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T17:54:57.307 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:54:57.307 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:54:57.318 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:54:57.318 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:54:58.551 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:54:58.556 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:54:58.557 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355698 2023-10-03T17:54:58.557 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355698 -le 1696358112 ]] 2023-10-03T17:54:58.557 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1789969421 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\7\8\9\9\6\9\4\2\1 ]] 2023-10-03T17:54:58.557 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:55:00.047 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.047 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.047 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.047 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c027d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c23a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.037+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b993580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c292780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b599a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.045+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724ad6e280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c23a080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.059+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b993580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.066+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.066+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.066+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.066+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c23a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c292780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.070 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b599a80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.071 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.067+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22c180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.079+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.079+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.079+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22c380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.079+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22c580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.084+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:55:00.085+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724b22cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:55:08.583 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:55:08.591 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:55:08.591 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:55:08.591 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:55:09.820 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:55:09.822 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:55:09.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355709 2023-10-03T17:55:09.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355709 -le 1696358112 ]] 2023-10-03T17:55:09.823 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1789969421 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\7\8\9\9\6\9\4\2\1 ]] 2023-10-03T17:55:09.824 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:55:19.829 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:55:19.829 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:55:19.842 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:55:19.863 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:55:21.068 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:55:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:55:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355721 2023-10-03T17:55:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355721 -le 1696358112 ]] 2023-10-03T17:55:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1789969421 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\7\8\9\9\6\9\4\2\1 ]] 2023-10-03T17:55:21.081 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:55:31.086 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:55:31.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:55:31.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:55:31.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:55:32.373 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1789969421 2023-10-03T17:55:32.373 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:55:32.380 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355732 2023-10-03T17:55:32.380 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355732 -le 1696358112 ]] 2023-10-03T17:55:32.380 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1789969421 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\7\8\9\9\6\9\4\2\1 ]] 2023-10-03T17:55:32.380 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:55:42.382 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:55:42.382 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:55:42.383 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:55:42.383 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:55:42.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:55:42.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.913+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:55:42.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.934+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.934+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.938+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.938+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.938+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.938+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:55:42.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:55:42.939+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:55:43.643 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:55:43.651 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:55:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355743 2023-10-03T17:55:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355743 -le 1696358112 ]] 2023-10-03T17:55:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\7\8\9\9\6\9\4\2\1 ]] 2023-10-03T17:55:43.655 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/1481624570 2023-10-03T17:55:46.310 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/1481624570 until 2023-10-03T18:55:45.627936+0000 (3600 sec) 2023-10-03T17:55:46.329 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:55:46.330 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/1481624570 2023-10-03T17:55:47.302 INFO:tasks.workunit.client.0.smithi031.stderr:listed 20 entries 2023-10-03T17:55:47.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:55:47.315 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:55:57.320 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:55:57.322 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:55:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:55:57.334 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:55:58.497 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:55:58.497 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:55:58.498 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355758 2023-10-03T17:55:58.498 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355758 -le 1696358112 ]] 2023-10-03T17:55:58.498 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:55:58.498 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:00.008 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.007+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.008 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.007+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b993d00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.029+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.030 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.029+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c23b100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.033 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.033+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.033 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.033+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22cf80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22cc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.039 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.038+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.107 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.107+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.107+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.108 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.109 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.109 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.108+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22db00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.132 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.132+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.132 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.132+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.138 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.138+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.138 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.138+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bc2a100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bc2a300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bc2a500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.139 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.139+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bc2a700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.160+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.160+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.161+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.161+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22d900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.161+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d780 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.161 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.161+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.170 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.170+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:00.171 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:56:00.171+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:56:08.511 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:56:08.515 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:56:08.523 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:56:08.546 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:56:09.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:56:09.816 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:56:09.816 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355769 2023-10-03T17:56:09.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355769 -le 1696358112 ]] 2023-10-03T17:56:09.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:56:09.817 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:14.526 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:14.526+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:56:19.832 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:56:19.838 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:56:19.838 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:56:19.844 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:56:21.009 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:56:21.009 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:56:21.010 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355781 2023-10-03T17:56:21.011 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355781 -le 1696358112 ]] 2023-10-03T17:56:21.011 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:56:21.011 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:31.034 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:56:31.035 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:56:31.035 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:56:31.035 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:56:32.298 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:56:32.300 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:56:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355792 2023-10-03T17:56:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355792 -le 1696358112 ]] 2023-10-03T17:56:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:56:32.301 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:38.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:38.535+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:56:38.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:38.599+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:56:38.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:38.599+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:56:38.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:38.603+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:56:42.315 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:56:42.315 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:56:42.316 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:56:42.351 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:56:43.581 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:56:43.582 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:56:43.583 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355803 2023-10-03T17:56:43.583 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355803 -le 1696358112 ]] 2023-10-03T17:56:43.583 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:56:43.583 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:51.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.597+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:56:51.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.606+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:56:51.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.610+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:56:51.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.613+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:56:51.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.613+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:56:51.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.613+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:56:51.684 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.682+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:56:51.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.682+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:56:51.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.682+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:56:51.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.684+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:56:51.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:56:51.684+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:56:53.586 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:56:53.586 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:56:53.591 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:56:53.601 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:56:54.878 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/1481624570 2023-10-03T17:56:54.878 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:56:54.878 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355814 2023-10-03T17:56:54.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355814 -le 1696358112 ]] 2023-10-03T17:56:54.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/1481624570 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:56:54.879 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:56:59.031 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:56:59.031+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T17:56:59.032 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T17:56:59.031+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T17:57:04.885 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:57:04.888 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:57:04.896 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:57:04.896 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:57:05.906 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:05.907 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:57:05.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355825 2023-10-03T17:57:05.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355825 -le 1696358112 ]] 2023-10-03T17:57:05.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/311399113 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\1\4\8\1\6\2\4\5\7\0 ]] 2023-10-03T17:57:05.908 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/311399113 2023-10-03T17:57:07.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:07.686+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:57:07.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:07.686+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:57:07.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:07.686+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:57:07.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:07.708+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:57:07.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:07.747+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:57:08.447 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/311399113 until 2023-10-03T18:57:07.699632+0000 (3600 sec) 2023-10-03T17:57:08.465 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:57:08.467 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/311399113 2023-10-03T17:57:09.336 INFO:tasks.workunit.client.0.smithi031.stderr:listed 21 entries 2023-10-03T17:57:09.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:09.344 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:57:19.350 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:57:19.350 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:57:19.350 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:57:19.351 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:57:20.502 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:20.507 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:57:20.509 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355840 2023-10-03T17:57:20.509 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355840 -le 1696358112 ]] 2023-10-03T17:57:20.509 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/311399113 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\1\3\9\9\1\1\3 ]] 2023-10-03T17:57:20.509 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:57:24.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:24.986+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:57:24.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:24.986+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:57:24.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:24.986+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:57:24.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:24.986+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:57:29.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.897+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:57:29.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.912+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:57:29.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.912+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:57:29.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.945+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:57:29.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.946+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:57:29.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:57:29.946+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:57:30.519 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:57:30.519 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:57:30.525 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:57:30.551 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:57:31.762 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:31.763 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:57:31.763 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355851 2023-10-03T17:57:31.763 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355851 -le 1696358112 ]] 2023-10-03T17:57:31.763 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/311399113 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\1\3\9\9\1\1\3 ]] 2023-10-03T17:57:31.763 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:57:41.773 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:57:41.773 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:57:41.776 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:57:41.783 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:57:42.945 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:42.962 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:57:42.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355862 2023-10-03T17:57:42.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355862 -le 1696358112 ]] 2023-10-03T17:57:42.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/311399113 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\1\3\9\9\1\1\3 ]] 2023-10-03T17:57:42.969 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:57:52.978 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:57:52.986 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:57:52.996 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:57:53.002 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:57:54.203 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/311399113 2023-10-03T17:57:54.205 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:57:54.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355874 2023-10-03T17:57:54.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355874 -le 1696358112 ]] 2023-10-03T17:57:54.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/311399113 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\1\3\9\9\1\1\3 ]] 2023-10-03T17:57:54.205 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:04.206 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:58:04.207 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:58:04.207 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:58:04.207 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:58:05.345 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:58:05.346 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355885 2023-10-03T17:58:05.347 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355885 -le 1696358112 ]] 2023-10-03T17:58:05.347 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\1\1\3\9\9\1\1\3 ]] 2023-10-03T17:58:05.347 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/2548654810 2023-10-03T17:58:06.859 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:06.859+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:58:06.882 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:06.882+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:58:07.368 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/2548654810 until 2023-10-03T18:58:06.626503+0000 (3600 sec) 2023-10-03T17:58:07.402 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/2548654810 2023-10-03T17:58:07.402 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:58:08.343 INFO:tasks.workunit.client.0.smithi031.stderr:listed 22 entries 2023-10-03T17:58:08.359 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:08.360 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:09.926 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:09.922+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T17:58:09.926 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:09.922+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T17:58:09.926 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:09.922+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T17:58:09.926 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:09.922+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:58:18.368 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:58:18.368 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:58:18.368 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:58:18.371 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:58:18.707 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T17:58:18.703+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T17:58:19.612 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:19.617 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:58:19.618 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355899 2023-10-03T17:58:19.618 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355899 -le 1696358112 ]] 2023-10-03T17:58:19.618 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:58:19.618 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:29.635 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:58:29.636 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:58:29.636 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:58:29.647 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:58:30.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:30.323+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:58:30.863 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:30.865 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:58:30.866 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355910 2023-10-03T17:58:30.867 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355910 -le 1696358112 ]] 2023-10-03T17:58:30.867 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:58:30.867 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:39.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.030+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:58:39.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.052+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:58:39.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.053+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:58:39.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.099+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:58:39.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.111+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:58:39.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:39.146+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:58:40.883 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:58:40.884 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:58:40.889 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:58:40.905 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:58:41.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.799+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:58:41.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.810+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:58:41.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.810+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:58:41.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.810+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:58:41.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.810+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:58:41.822 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.816+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:58:41.822 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:41.816+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:58:42.051 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:42.051 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:58:42.052 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355922 2023-10-03T17:58:42.052 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355922 -le 1696358112 ]] 2023-10-03T17:58:42.052 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:58:42.052 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:52.058 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:58:52.068 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:58:52.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:58:52.087 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:58:53.156 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:58:53.156 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:58:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355933 2023-10-03T17:58:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355933 -le 1696358112 ]] 2023-10-03T17:58:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:58:53.157 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:58:55.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.278+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:58:55.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.278+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:58:55.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.285+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:58:55.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.285+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:58:55.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.314+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:58:55.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.314+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:58:55.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.314+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:58:55.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.314+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:58:55.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:55.320+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:58:56.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.093+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:58:56.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.093+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:58:56.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.176+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:58:56.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.176+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:58:56.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.176+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:58:56.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:58:56.176+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:59:00.010 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.002+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.010 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.002+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c327a00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.011 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.007+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.011 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.007+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c027500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.011 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.008+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.011 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.008+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c08b380 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c332680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c2e8180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.027 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.025+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724bc2b900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.028+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22dd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.028+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724b22de80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.033 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.033+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.034 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.033+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62a680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.044 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.036+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.045 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.036+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62a800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.053 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.051+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62a280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.054+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.055+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.055+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.055+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62a980 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.055+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62ab80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.063 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.055+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62ad80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.064 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.062+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.064 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.062+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62af80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.079 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62b180 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62bb80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:00.080 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T17:59:00.076+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62ba00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T17:59:02.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.101+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T17:59:02.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.193+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:59:02.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.193+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:59:02.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.199+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:59:02.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.199+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:59:02.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.204+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:59:02.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.205+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:59:02.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.205+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:59:02.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.205+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:59:02.208 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.205+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:59:02.208 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:02.205+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:59:03.169 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:59:03.169 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:59:03.188 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:59:03.216 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:59:04.349 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/2548654810 2023-10-03T17:59:04.371 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:59:04.371 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355944 2023-10-03T17:59:04.372 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355944 -le 1696358112 ]] 2023-10-03T17:59:04.372 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/2548654810 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:59:04.372 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:59:14.384 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:59:14.385 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:59:14.385 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:59:14.386 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:59:15.534 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T17:59:15.534 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:59:15.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355955 2023-10-03T17:59:15.535 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355955 -le 1696358112 ]] 2023-10-03T17:59:15.536 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\2\5\4\8\6\5\4\8\1\0 ]] 2023-10-03T17:59:15.536 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/3260346930 2023-10-03T17:59:17.493 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/3260346930 until 2023-10-03T18:59:16.757732+0000 (3600 sec) 2023-10-03T17:59:17.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/3260346930 2023-10-03T17:59:17.516 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T17:59:18.621 INFO:tasks.workunit.client.0.smithi031.stderr:listed 23 entries 2023-10-03T17:59:18.644 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T17:59:18.645 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:59:24.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.109+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:59:24.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.109+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:59:24.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.109+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:59:24.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.109+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:59:24.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.141+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:59:24.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:24.141+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:59:28.647 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:59:28.652 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:59:28.652 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:59:28.673 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:59:29.870 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T17:59:29.870 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:59:29.871 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355969 2023-10-03T17:59:29.871 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355969 -le 1696358112 ]] 2023-10-03T17:59:29.871 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T17:59:29.871 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:59:32.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.201+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:59:32.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.213+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:59:32.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.266+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:59:32.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.266+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:59:32.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.266+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:59:32.283 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.275+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:59:32.283 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:32.280+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:59:39.880 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:59:39.884 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:59:39.935 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:59:39.941 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:59:41.168 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T17:59:41.168 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:59:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355981 2023-10-03T17:59:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355981 -le 1696358112 ]] 2023-10-03T17:59:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T17:59:41.169 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:59:42.254 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:42.237+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:59:42.254 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:42.237+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:59:42.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:42.287+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T17:59:51.177 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T17:59:51.178 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T17:59:51.178 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T17:59:51.199 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T17:59:52.365 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T17:59:52.366 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T17:59:52.367 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696355992 2023-10-03T17:59:52.367 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696355992 -le 1696358112 ]] 2023-10-03T17:59:52.367 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T17:59:52.367 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T17:59:53.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.781+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T17:59:53.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T17:59:53.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T17:59:53.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T17:59:53.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.809+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T17:59:53.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.809+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T17:59:53.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.809+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T17:59:53.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.865+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T17:59:53.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.865+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T17:59:53.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T17:59:53.889+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:00:00.012 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.012 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.012 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.012 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62bc80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.012 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724c62bb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.013 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.010+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c62be80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.013 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.011+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.013 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.011+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbe080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.057 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbe280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbe480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbe680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbe880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.055+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbea80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.082 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbee80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbf080 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbf280 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.078+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbf480 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.083 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbf680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbfe80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbfd00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbfb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.081+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724afbf880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.085+0000 7f57f90f0700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:00.089 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:00:00.085+0000 7f57f90f0700 -1 librbd::ImageState: 0x55724afbec80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:00:02.369 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:00:02.371 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:00:02.379 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:00:02.388 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:00:03.516 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T18:00:03.518 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:00:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356003 2023-10-03T18:00:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356003 -le 1696358112 ]] 2023-10-03T18:00:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T18:00:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:00:08.239 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.237+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:00:08.239 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.237+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:00:08.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.270+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:00:08.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.364+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:00:08.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.364+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:00:08.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.364+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:00:08.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:08.364+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:00:09.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:09.602+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:00:09.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:09.602+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:00:09.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:09.602+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:00:13.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:00:13.521 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:00:13.522 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:00:13.523 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:00:14.709 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/3260346930 2023-10-03T18:00:14.710 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:00:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356014 2023-10-03T18:00:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356014 -le 1696358112 ]] 2023-10-03T18:00:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/3260346930 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T18:00:14.712 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:00:24.717 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:00:24.717 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:00:24.726 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:00:24.751 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:00:25.898 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:00:25.898 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:00:25.899 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356025 2023-10-03T18:00:25.899 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356025 -le 1696358112 ]] 2023-10-03T18:00:25.900 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/874826947 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\3\2\6\0\3\4\6\9\3\0 ]] 2023-10-03T18:00:25.900 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 172.21.15.31:0/874826947 2023-10-03T18:00:28.458 INFO:tasks.workunit.client.0.smithi031.stderr:blocklisting 172.21.15.31:0/874826947 until 2023-10-03T19:00:27.727101+0000 (3600 sec) 2023-10-03T18:00:28.480 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist ls 2023-10-03T18:00:28.481 INFO:tasks.workunit.client.0.smithi031.stderr:+ grep -q 172.21.15.31:0/874826947 2023-10-03T18:00:28.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:28.680+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:00:28.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:28.686+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:00:29.456 INFO:tasks.workunit.client.0.smithi031.stderr:listed 24 entries 2023-10-03T18:00:29.472 INFO:tasks.workunit.client.0.smithi031.stderr:+ PREV_CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:00:29.472 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:00:30.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:30.827+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:00:31.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.016+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:00:31.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.016+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:00:31.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:00:31.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.024+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:00:31.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.024+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:00:31.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.063+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:00:31.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.063+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:00:31.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.063+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:00:31.087 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.081+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:00:31.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:00:31.091+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:00:39.478 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:00:39.500 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:00:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:00:39.515 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:00:41.006 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:00:41.007 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:00:41.007 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356041 2023-10-03T18:00:41.008 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356041 -le 1696358112 ]] 2023-10-03T18:00:41.008 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/874826947 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\4\8\2\6\9\4\7 ]] 2023-10-03T18:00:41.008 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:00:44.358 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:00:44.358+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:00:44.358 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:00:44.358+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:00:44.358 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:00:44.358+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:00:51.012 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:00:51.013 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:00:51.017 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:00:51.017 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:00:52.267 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:00:52.272 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:00:52.273 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356052 2023-10-03T18:00:52.273 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356052 -le 1696358112 ]] 2023-10-03T18:00:52.273 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/874826947 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\4\8\2\6\9\4\7 ]] 2023-10-03T18:00:52.273 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bb1e300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b79fb00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.029 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.026+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c332200 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.048 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b79e700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c333000 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724bc2ae80 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.049 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.047+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22d500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.058 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b947580 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c62b800 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c2e8680 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.059 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.057+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c277e00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.059+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.060 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.059+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724b22c500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.070+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.070+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724afbf880 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.072+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.073 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.072+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724afbf700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.074+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.084 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.074+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c4d4100 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.091 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.090+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.091 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.090+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.091 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c4d4300 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.091 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.090+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c4d4500 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f88ef700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f80ee700 -1 librbd::image::OpenRequest: failed to retrieve name: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c4d4700 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f80ee700 -1 librbd::ImageState: 0x55724c4d4900 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:00.098 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:01:00.096+0000 7f57f88ef700 -1 librbd::ImageState: 0x55724c4d4b00 failed to open image: (108) Cannot send after transport endpoint shutdown 2023-10-03T18:01:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:01:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:01:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:01:02.285 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:01:03.515 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:01:03.519 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:01:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356063 2023-10-03T18:01:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356063 -le 1696358112 ]] 2023-10-03T18:01:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/874826947 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\4\8\2\6\9\4\7 ]] 2023-10-03T18:01:03.521 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:01:13.527 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:01:13.528 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:01:13.528 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:01:13.534 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:01:14.773 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR=172.21.15.31:0/874826947 2023-10-03T18:01:14.780 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:01:14.780 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356074 2023-10-03T18:01:14.780 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356074 -le 1696358112 ]] 2023-10-03T18:01:14.781 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 172.21.15.31:0/874826947 != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\4\8\2\6\9\4\7 ]] 2023-10-03T18:01:14.781 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 10 2023-10-03T18:01:17.902 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:17.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:01:17.927 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:17.925+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:01:17.949 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:17.944+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:01:21.127 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.119+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:01:21.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.125+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:01:21.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.125+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:01:21.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.151+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:01:21.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:01:21.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:21.151+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:01:24.783 INFO:tasks.workunit.client.0.smithi031.stderr:++ ceph mgr dump 2023-10-03T18:01:24.800 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq '.active_clients[]' 2023-10-03T18:01:24.800 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq 'select(.name == "rbd_support")' 2023-10-03T18:01:24.801 INFO:tasks.workunit.client.0.smithi031.stderr:++ jq -r '[.addrvec[0].addr, "/", .addrvec[0].nonce|tostring] | add' 2023-10-03T18:01:25.925 INFO:tasks.workunit.client.0.smithi031.stderr:+ CLIENT_ADDR= 2023-10-03T18:01:25.925 INFO:tasks.workunit.client.0.smithi031.stderr:++ date +%s 2023-10-03T18:01:25.926 INFO:tasks.workunit.client.0.smithi031.stderr:+ CURRENT_TIME=1696356085 2023-10-03T18:01:25.926 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ 1696356085 -le 1696358112 ]] 2023-10-03T18:01:25.927 INFO:tasks.workunit.client.0.smithi031.stderr:+ [[ '' != \1\7\2\.\2\1\.\1\5\.\3\1\:\0\/\8\7\4\8\2\6\9\4\7 ]] 2023-10-03T18:01:25.927 INFO:tasks.workunit.client.0.smithi031.stderr:+ ceph osd blocklist add 2023-10-03T18:01:26.591 INFO:tasks.workunit.client.0.smithi031.stderr:Invalid command: missing required parameter addr() 2023-10-03T18:01:26.591 INFO:tasks.workunit.client.0.smithi031.stderr:osd blocklist [] [] : add (optionally until seconds from now) or remove from blocklist 2023-10-03T18:01:26.591 INFO:tasks.workunit.client.0.smithi031.stderr:Error EINVAL: invalid command 2023-10-03T18:01:26.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-03T18:01:26.616 INFO:tasks.workunit:Stopping ['rbd/rbd_support_module_recovery.sh'] on client.0... 2023-10-03T18:01:26.617 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-03T18:01:27.348 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi031 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T18:01:27.417 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=aabfa6783e8e4e5a9628900ae137d4a1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi031 with status 22: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh' 2023-10-03T18:01:27.420 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T18:01:27.467 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-03T18:01:27.475 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-03T18:01:27.475 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:01:27.646 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:27.646+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:01:27.683 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:27.673+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:01:28.285 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:28.323 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:01:28.340 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":940,"stamp":"2023-10-03T18:01:26.398281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21855869934,"num_objects":10329,"num_object_clones":5120,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":854634,"num_read_kb":3240760,"num_write":829154,"num_write_kb":3317071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80510,"ondisk_log_size":80510,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9631664,"kb_used_data":6633240,"kb_used_omap":364,"kb_used_meta":2998035,"kb_avail":273483856,"statfs":{"total":289910292480,"available":280047468544,"internally_reserved":0,"allocated":6792437760,"data_stored":12549876747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3069988287},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,6,15,19,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22073344,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5584,"num_read_kb":22220,"num_write":5006,"num_write_kb":20024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013372"},"pg_stats":[{"pgid":"2.7","version":"88'106564","reported_seq":211103,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:21.751151+0000","last_change":"2023-10-03T17:34:48.146004+0000","last_active":"2023-10-03T18:01:21.751151+0000","last_peered":"2023-10-03T18:01:21.751151+0000","last_clean":"2023-10-03T18:01:21.751151+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:01:21.751151+0000","last_undegraded":"2023-10-03T18:01:21.751151+0000","last_fullsized":"2023-10-03T18:01:21.751151+0000","mapping_epoch":12,"log_start":"84'96508","ondisk_log_start":"84'96508","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:23.665197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2724405531,"num_objects":1285,"num_object_clones":638,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":105830,"num_read_kb":404672,"num_write":103270,"num_write_kb":413017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'97141","reported_seq":193952,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:21.757536+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:01:21.757536+0000","last_peered":"2023-10-03T18:01:21.757536+0000","last_clean":"2023-10-03T18:01:21.757536+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:01:21.757536+0000","last_undegraded":"2023-10-03T18:01:21.757536+0000","last_fullsized":"2023-10-03T18:01:21.757536+0000","mapping_epoch":12,"log_start":"84'87104","ondisk_log_start":"84'87104","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:03:34.306073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2515300570,"num_objects":1185,"num_object_clones":589,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":98939,"num_read_kb":373801,"num_write":95220,"num_write_kb":380706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'117719","reported_seq":231067,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:21.740058+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:01:21.740058+0000","last_peered":"2023-10-03T18:01:21.740058+0000","last_clean":"2023-10-03T18:01:21.740058+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:01:21.740058+0000","last_undegraded":"2023-10-03T18:01:21.740058+0000","last_fullsized":"2023-10-03T18:01:21.740058+0000","mapping_epoch":12,"log_start":"84'107705","ondisk_log_start":"84'107705","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:44:27.523925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2949087583,"num_objects":1395,"num_object_clones":691,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":115468,"num_read_kb":435902,"num_write":111740,"num_write_kb":446849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"88'110570","reported_seq":219563,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:23.119638+0000","last_change":"2023-10-03T17:34:48.152806+0000","last_active":"2023-10-03T18:01:23.119638+0000","last_peered":"2023-10-03T18:01:23.119638+0000","last_clean":"2023-10-03T18:01:23.119638+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:01:23.119638+0000","last_undegraded":"2023-10-03T18:01:23.119638+0000","last_fullsized":"2023-10-03T18:01:23.119638+0000","mapping_epoch":12,"log_start":"84'100506","ondisk_log_start":"84'100506","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:50:32.001473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2851360969,"num_objects":1344,"num_object_clones":668,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":110322,"num_read_kb":421736,"num_write":108656,"num_write_kb":434564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":228,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-10-03T18:00:27.483434+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T18:00:27.483434+0000","last_peered":"2023-10-03T18:00:27.483434+0000","last_clean":"2023-10-03T18:00:27.483434+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:00:27.483434+0000","last_undegraded":"2023-10-03T18:00:27.483434+0000","last_fullsized":"2023-10-03T18:00:27.483434+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'112874","reported_seq":224007,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:21.736420+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:01:21.736420+0000","last_peered":"2023-10-03T18:01:21.736420+0000","last_clean":"2023-10-03T18:01:21.736420+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:01:21.736420+0000","last_undegraded":"2023-10-03T18:01:21.736420+0000","last_fullsized":"2023-10-03T18:01:21.736420+0000","mapping_epoch":12,"log_start":"84'102806","ondisk_log_start":"84'102806","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:32:01.804277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2897797505,"num_objects":1369,"num_object_clones":679,"num_object_copies":2738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":112747,"num_read_kb":430284,"num_write":109527,"num_write_kb":438030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'102356","reported_seq":203362,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:23.141871+0000","last_change":"2023-10-03T17:34:48.152786+0000","last_active":"2023-10-03T18:01:23.141871+0000","last_peered":"2023-10-03T18:01:23.141871+0000","last_clean":"2023-10-03T18:01:23.141871+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:01:23.141871+0000","last_undegraded":"2023-10-03T18:01:23.141871+0000","last_fullsized":"2023-10-03T18:01:23.141871+0000","mapping_epoch":12,"log_start":"84'92306","ondisk_log_start":"84'92306","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:12.166372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2640703757,"num_objects":1247,"num_object_clones":619,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":102657,"num_read_kb":389197,"num_write":99878,"num_write_kb":399431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"88'102847","reported_seq":206270,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:24.378942+0000","last_change":"2023-10-03T17:34:48.140895+0000","last_active":"2023-10-03T18:01:24.378942+0000","last_peered":"2023-10-03T18:01:24.378942+0000","last_clean":"2023-10-03T18:01:24.378942+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:01:24.378942+0000","last_undegraded":"2023-10-03T18:01:24.378942+0000","last_fullsized":"2023-10-03T18:01:24.378942+0000","mapping_epoch":12,"log_start":"84'92805","ondisk_log_start":"84'92805","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:56:30.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2644500951,"num_objects":1255,"num_object_clones":619,"num_object_copies":2510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":105734,"num_read_kb":395151,"num_write":101029,"num_write_kb":403924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"88'102987","reported_seq":204505,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:21.757831+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:01:21.757831+0000","last_peered":"2023-10-03T18:01:21.757831+0000","last_clean":"2023-10-03T18:01:21.757831+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:01:21.757831+0000","last_undegraded":"2023-10-03T18:01:21.757831+0000","last_fullsized":"2023-10-03T18:01:21.757831+0000","mapping_epoch":12,"log_start":"84'92905","ondisk_log_start":"84'92905","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:54:42.067523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2632122700,"num_objects":1247,"num_object_clones":617,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":102831,"num_read_kb":389926,"num_write":99697,"num_write_kb":398656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21855279566,"num_objects":10327,"num_object_clones":5120,"num_object_copies":20654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":854528,"num_read_kb":3240669,"num_write":829017,"num_write_kb":3315177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6791204864,"data_stored":12549915548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80413,"ondisk_log_size":80413,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738713,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2463628,"kb_used_data":1272644,"kb_used_omap":1,"kb_used_meta":1190974,"kb_avail":91908212,"statfs":{"total":96636764160,"available":94114009088,"internally_reserved":0,"allocated":1303187456,"data_stored":2408055996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1219557830},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.9689999999999999,"15min":2.673},"min":{"1min":0.219,"5min":0.187,"15min":0.16600000000000001},"max":{"1min":10.962999999999999,"5min":18.648,"15min":22.562000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":4.2439999999999998,"5min":3.222,"15min":2.738},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.472,"5min":18.041,"15min":21.827999999999999},"last":7.6269999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:00:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1440000000000001,"5min":2.8719999999999999,"15min":2.4740000000000002},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":11.007,"5min":26.074000000000002,"15min":26.074000000000002},"last":7.516},{"interface":"front","average":{"1min":2.1960000000000002,"5min":2.0590000000000002,"15min":1.946},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.188},"max":{"1min":16.457000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":1.746}]}]},{"osd":0,"up_from":8,"seq":34359738713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2630096,"kb_used_data":2046344,"kb_used_omap":347,"kb_used_meta":583396,"kb_avail":91741744,"statfs":{"total":96636764160,"available":93943545856,"internally_reserved":0,"allocated":2095456256,"data_stored":3869096444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":597397680},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":3.766,"15min":3.4700000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.143000000000001,"5min":21.928999999999998,"15min":21.928999999999998},"last":1.714},{"interface":"front","average":{"1min":3.012,"5min":3.02,"15min":3.6309999999999998},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.19},"max":{"1min":11.896000000000001,"5min":18.492000000000001,"15min":24.215},"last":1.675}]},{"osd":2,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.2879999999999998,"15min":2.4399999999999999},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.9880000000000004,"5min":12.256,"15min":12.256},"last":2.1259999999999999},{"interface":"front","average":{"1min":2.4260000000000002,"5min":2.3029999999999999,"15min":2.3050000000000002},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":11.94,"5min":12.630000000000001,"15min":13.677},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738708,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4537940,"kb_used_data":3314252,"kb_used_omap":15,"kb_used_meta":1223664,"kb_avail":89833900,"statfs":{"total":96636764160,"available":91989913600,"internally_reserved":0,"allocated":3393794048,"data_stored":6272724307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1253032777},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,6,5,6,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.4050000000000002,"5min":3.4780000000000002,"15min":3.8130000000000002},"min":{"1min":0.33200000000000002,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":12.513999999999999,"5min":14.651,"15min":39.990000000000002},"last":0.17100000000000001},{"interface":"front","average":{"1min":6.2430000000000003,"5min":3.9820000000000002,"15min":3.8380000000000001},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":20.716999999999999,"5min":24.481000000000002,"15min":37.381},"last":8.6590000000000007}]},{"osd":2,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.9199999999999999,"5min":3.1669999999999998,"15min":2.3740000000000001},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.17299999999999999},"max":{"1min":24.579000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":5.9580000000000002},{"interface":"front","average":{"1min":1.873,"5min":2.464,"15min":3.48},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.6029999999999998,"5min":18.677,"15min":22.739000000000001},"last":0.22500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2094903296,"data_stored":3868968567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3393323008,"data_stored":6272678350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1302978560,"data_stored":2408268631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:01:28.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:01:29.039 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:29.088 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:01:29.127 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":941,"stamp":"2023-10-03T18:01:28.398735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21865319406,"num_objects":10329,"num_object_clones":5120,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":857666,"num_read_kb":3250933,"num_write":831236,"num_write_kb":3325399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80551,"ondisk_log_size":80551,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9674840,"kb_used_data":6646272,"kb_used_omap":364,"kb_used_meta":3028179,"kb_avail":273440680,"statfs":{"total":289910292480,"available":280003256320,"internally_reserved":0,"allocated":6805782528,"data_stored":12563221515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3100855743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,0,15,16,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31522816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8616,"num_read_kb":32393,"num_write":7088,"num_write_kb":28352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013481"},"pg_stats":[{"pgid":"2.7","version":"88'106864","reported_seq":211764,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.831925+0000","last_change":"2023-10-03T17:34:48.146004+0000","last_active":"2023-10-03T18:01:26.831925+0000","last_peered":"2023-10-03T18:01:26.831925+0000","last_clean":"2023-10-03T18:01:26.831925+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:01:26.831925+0000","last_undegraded":"2023-10-03T18:01:26.831925+0000","last_fullsized":"2023-10-03T18:01:26.831925+0000","mapping_epoch":12,"log_start":"84'96808","ondisk_log_start":"84'96808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:23.665197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2725708059,"num_objects":1285,"num_object_clones":638,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":106245,"num_read_kb":406090,"num_write":103562,"num_write_kb":414185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'97438","reported_seq":194599,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.798762+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:01:26.798762+0000","last_peered":"2023-10-03T18:01:26.798762+0000","last_clean":"2023-10-03T18:01:26.798762+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:01:26.798762+0000","last_undegraded":"2023-10-03T18:01:26.798762+0000","last_fullsized":"2023-10-03T18:01:26.798762+0000","mapping_epoch":12,"log_start":"84'87404","ondisk_log_start":"84'87404","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:03:34.306073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2516947162,"num_objects":1185,"num_object_clones":589,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":99317,"num_read_kb":375105,"num_write":95513,"num_write_kb":381878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'118084","reported_seq":231850,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.786020+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:01:26.786020+0000","last_peered":"2023-10-03T18:01:26.786020+0000","last_clean":"2023-10-03T18:01:26.786020+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:01:26.786020+0000","last_undegraded":"2023-10-03T18:01:26.786020+0000","last_fullsized":"2023-10-03T18:01:26.786020+0000","mapping_epoch":12,"log_start":"84'108005","ondisk_log_start":"84'108005","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:44:27.523925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2950570335,"num_objects":1395,"num_object_clones":691,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":115965,"num_read_kb":437527,"num_write":112089,"num_write_kb":448245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"88'110877","reported_seq":220252,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:28.195114+0000","last_change":"2023-10-03T17:34:48.152806+0000","last_active":"2023-10-03T18:01:28.195114+0000","last_peered":"2023-10-03T18:01:28.195114+0000","last_clean":"2023-10-03T18:01:28.195114+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:01:28.195114+0000","last_undegraded":"2023-10-03T18:01:28.195114+0000","last_fullsized":"2023-10-03T18:01:28.195114+0000","mapping_epoch":12,"log_start":"84'100806","ondisk_log_start":"84'100806","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:50:32.001473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2852655305,"num_objects":1344,"num_object_clones":668,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":110764,"num_read_kb":423226,"num_write":108957,"num_write_kb":435768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":229,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:00:28.459283+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T18:00:28.459283+0000","last_peered":"2023-10-03T18:00:28.459283+0000","last_clean":"2023-10-03T18:00:28.459283+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:00:28.459283+0000","last_undegraded":"2023-10-03T18:00:28.459283+0000","last_fullsized":"2023-10-03T18:00:28.459283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'113205","reported_seq":224742,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.799345+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:01:26.799345+0000","last_peered":"2023-10-03T18:01:26.799345+0000","last_clean":"2023-10-03T18:01:26.799345+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:01:26.799345+0000","last_undegraded":"2023-10-03T18:01:26.799345+0000","last_fullsized":"2023-10-03T18:01:26.799345+0000","mapping_epoch":12,"log_start":"84'103106","ondisk_log_start":"84'103106","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:32:01.804277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2899210625,"num_objects":1369,"num_object_clones":679,"num_object_copies":2738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":113207,"num_read_kb":431847,"num_write":109850,"num_write_kb":439322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'102636","reported_seq":204004,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:28.188998+0000","last_change":"2023-10-03T17:34:48.152786+0000","last_active":"2023-10-03T18:01:28.188998+0000","last_peered":"2023-10-03T18:01:28.188998+0000","last_clean":"2023-10-03T18:01:28.188998+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:01:28.188998+0000","last_undegraded":"2023-10-03T18:01:28.188998+0000","last_fullsized":"2023-10-03T18:01:28.188998+0000","mapping_epoch":12,"log_start":"84'92606","ondisk_log_start":"84'92606","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:12.166372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2641944845,"num_objects":1247,"num_object_clones":619,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103104,"num_read_kb":390582,"num_write":100149,"num_write_kb":400515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"88'102847","reported_seq":206270,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:24.378942+0000","last_change":"2023-10-03T17:34:48.140895+0000","last_active":"2023-10-03T18:01:24.378942+0000","last_peered":"2023-10-03T18:01:24.378942+0000","last_clean":"2023-10-03T18:01:24.378942+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:01:24.378942+0000","last_undegraded":"2023-10-03T18:01:24.378942+0000","last_fullsized":"2023-10-03T18:01:24.378942+0000","mapping_epoch":12,"log_start":"84'92805","ondisk_log_start":"84'92805","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:56:30.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2644500951,"num_objects":1255,"num_object_clones":619,"num_object_copies":2510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":105734,"num_read_kb":395151,"num_write":101029,"num_write_kb":403924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"88'103248","reported_seq":205136,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.772413+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:01:26.772413+0000","last_peered":"2023-10-03T18:01:26.772413+0000","last_clean":"2023-10-03T18:01:26.772413+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:01:26.772413+0000","last_undegraded":"2023-10-03T18:01:26.772413+0000","last_fullsized":"2023-10-03T18:01:26.772413+0000","mapping_epoch":12,"log_start":"84'93205","ondisk_log_start":"84'93205","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:54:42.067523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2633191756,"num_objects":1247,"num_object_clones":617,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103224,"num_read_kb":391314,"num_write":99950,"num_write_kb":399668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21864729038,"num_objects":10327,"num_object_clones":5120,"num_object_copies":20654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":857560,"num_read_kb":3250842,"num_write":831099,"num_write_kb":3323505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6804480000,"data_stored":12563190684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80454,"ondisk_log_size":80454,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738714,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2467172,"kb_used_data":1276188,"kb_used_omap":1,"kb_used_meta":1190974,"kb_avail":91904668,"statfs":{"total":96636764160,"available":94110380032,"internally_reserved":0,"allocated":1306816512,"data_stored":2411685052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1219557830},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.9689999999999999,"15min":2.673},"min":{"1min":0.219,"5min":0.187,"15min":0.16600000000000001},"max":{"1min":10.962999999999999,"5min":18.648,"15min":22.562000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":4.2439999999999998,"5min":3.222,"15min":2.738},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.472,"5min":18.041,"15min":21.827999999999999},"last":7.6269999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:00:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1440000000000001,"5min":2.8719999999999999,"15min":2.4740000000000002},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":11.007,"5min":26.074000000000002,"15min":26.074000000000002},"last":7.516},{"interface":"front","average":{"1min":2.1960000000000002,"5min":2.0590000000000002,"15min":1.946},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.188},"max":{"1min":16.457000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":1.746}]}]},{"osd":0,"up_from":8,"seq":34359738713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2630096,"kb_used_data":2046344,"kb_used_omap":347,"kb_used_meta":583396,"kb_avail":91741744,"statfs":{"total":96636764160,"available":93943545856,"internally_reserved":0,"allocated":2095456256,"data_stored":3869096444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":597397680},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":3.766,"15min":3.4700000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.143000000000001,"5min":21.928999999999998,"15min":21.928999999999998},"last":1.714},{"interface":"front","average":{"1min":3.012,"5min":3.02,"15min":3.6309999999999998},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.19},"max":{"1min":11.896000000000001,"5min":18.492000000000001,"15min":24.215},"last":1.675}]},{"osd":2,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.2879999999999998,"15min":2.4399999999999999},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.9880000000000004,"5min":12.256,"15min":12.256},"last":2.1259999999999999},{"interface":"front","average":{"1min":2.4260000000000002,"5min":2.3029999999999999,"15min":2.3050000000000002},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":11.94,"5min":12.630000000000001,"15min":13.677},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738709,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4577572,"kb_used_data":3323740,"kb_used_omap":15,"kb_used_meta":1253808,"kb_avail":89794268,"statfs":{"total":96636764160,"available":91949330432,"internally_reserved":0,"allocated":3403509760,"data_stored":6282440019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1283900233},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,0,14,7,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.4050000000000002,"5min":3.4780000000000002,"15min":3.8130000000000002},"min":{"1min":0.33200000000000002,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":12.513999999999999,"5min":14.651,"15min":39.990000000000002},"last":4.79},{"interface":"front","average":{"1min":6.2430000000000003,"5min":3.9820000000000002,"15min":3.8380000000000001},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":20.716999999999999,"5min":24.481000000000002,"15min":37.381},"last":4.8230000000000004}]},{"osd":2,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.9199999999999999,"5min":3.1669999999999998,"15min":2.3740000000000001},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.17299999999999999},"max":{"1min":24.579000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":1.873,"5min":2.464,"15min":3.48},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.6029999999999998,"5min":18.677,"15min":22.739000000000001},"last":3.2549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2094903296,"data_stored":3868968567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3403059200,"data_stored":6282414542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1306517504,"data_stored":2411807575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:01:29.128 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-03T18:01:29.129 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:01:29.822 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:29.825 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:01:29.845 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":941,"stamp":"2023-10-03T18:01:28.398735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21865319406,"num_objects":10329,"num_object_clones":5120,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":857666,"num_read_kb":3250933,"num_write":831236,"num_write_kb":3325399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80551,"ondisk_log_size":80551,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9674840,"kb_used_data":6646272,"kb_used_omap":364,"kb_used_meta":3028179,"kb_avail":273440680,"statfs":{"total":289910292480,"available":280003256320,"internally_reserved":0,"allocated":6805782528,"data_stored":12563221515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3100855743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,0,15,16,13,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31522816,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8616,"num_read_kb":32393,"num_write":7088,"num_write_kb":28352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.013481"},"pg_stats":[{"pgid":"2.7","version":"88'106864","reported_seq":211764,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.831925+0000","last_change":"2023-10-03T17:34:48.146004+0000","last_active":"2023-10-03T18:01:26.831925+0000","last_peered":"2023-10-03T18:01:26.831925+0000","last_clean":"2023-10-03T18:01:26.831925+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:01:26.831925+0000","last_undegraded":"2023-10-03T18:01:26.831925+0000","last_fullsized":"2023-10-03T18:01:26.831925+0000","mapping_epoch":12,"log_start":"84'96808","ondisk_log_start":"84'96808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:23.665197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2725708059,"num_objects":1285,"num_object_clones":638,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":106245,"num_read_kb":406090,"num_write":103562,"num_write_kb":414185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'97438","reported_seq":194599,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.798762+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:01:26.798762+0000","last_peered":"2023-10-03T18:01:26.798762+0000","last_clean":"2023-10-03T18:01:26.798762+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:01:26.798762+0000","last_undegraded":"2023-10-03T18:01:26.798762+0000","last_fullsized":"2023-10-03T18:01:26.798762+0000","mapping_epoch":12,"log_start":"84'87404","ondisk_log_start":"84'87404","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:03:34.306073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2516947162,"num_objects":1185,"num_object_clones":589,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":99317,"num_read_kb":375105,"num_write":95513,"num_write_kb":381878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'118084","reported_seq":231850,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.786020+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:01:26.786020+0000","last_peered":"2023-10-03T18:01:26.786020+0000","last_clean":"2023-10-03T18:01:26.786020+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:01:26.786020+0000","last_undegraded":"2023-10-03T18:01:26.786020+0000","last_fullsized":"2023-10-03T18:01:26.786020+0000","mapping_epoch":12,"log_start":"84'108005","ondisk_log_start":"84'108005","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:44:27.523925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2950570335,"num_objects":1395,"num_object_clones":691,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":115965,"num_read_kb":437527,"num_write":112089,"num_write_kb":448245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"88'110877","reported_seq":220252,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:28.195114+0000","last_change":"2023-10-03T17:34:48.152806+0000","last_active":"2023-10-03T18:01:28.195114+0000","last_peered":"2023-10-03T18:01:28.195114+0000","last_clean":"2023-10-03T18:01:28.195114+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:01:28.195114+0000","last_undegraded":"2023-10-03T18:01:28.195114+0000","last_fullsized":"2023-10-03T18:01:28.195114+0000","mapping_epoch":12,"log_start":"84'100806","ondisk_log_start":"84'100806","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:50:32.001473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2852655305,"num_objects":1344,"num_object_clones":668,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":110764,"num_read_kb":423226,"num_write":108957,"num_write_kb":435768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":229,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:00:28.459283+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T18:00:28.459283+0000","last_peered":"2023-10-03T18:00:28.459283+0000","last_clean":"2023-10-03T18:00:28.459283+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:00:28.459283+0000","last_undegraded":"2023-10-03T18:00:28.459283+0000","last_fullsized":"2023-10-03T18:00:28.459283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'113205","reported_seq":224742,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.799345+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:01:26.799345+0000","last_peered":"2023-10-03T18:01:26.799345+0000","last_clean":"2023-10-03T18:01:26.799345+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:01:26.799345+0000","last_undegraded":"2023-10-03T18:01:26.799345+0000","last_fullsized":"2023-10-03T18:01:26.799345+0000","mapping_epoch":12,"log_start":"84'103106","ondisk_log_start":"84'103106","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:32:01.804277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2899210625,"num_objects":1369,"num_object_clones":679,"num_object_copies":2738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":113207,"num_read_kb":431847,"num_write":109850,"num_write_kb":439322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'102636","reported_seq":204004,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:28.188998+0000","last_change":"2023-10-03T17:34:48.152786+0000","last_active":"2023-10-03T18:01:28.188998+0000","last_peered":"2023-10-03T18:01:28.188998+0000","last_clean":"2023-10-03T18:01:28.188998+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:01:28.188998+0000","last_undegraded":"2023-10-03T18:01:28.188998+0000","last_fullsized":"2023-10-03T18:01:28.188998+0000","mapping_epoch":12,"log_start":"84'92606","ondisk_log_start":"84'92606","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:12.166372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2641944845,"num_objects":1247,"num_object_clones":619,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103104,"num_read_kb":390582,"num_write":100149,"num_write_kb":400515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"88'102847","reported_seq":206270,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:24.378942+0000","last_change":"2023-10-03T17:34:48.140895+0000","last_active":"2023-10-03T18:01:24.378942+0000","last_peered":"2023-10-03T18:01:24.378942+0000","last_clean":"2023-10-03T18:01:24.378942+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:01:24.378942+0000","last_undegraded":"2023-10-03T18:01:24.378942+0000","last_fullsized":"2023-10-03T18:01:24.378942+0000","mapping_epoch":12,"log_start":"84'92805","ondisk_log_start":"84'92805","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:56:30.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2644500951,"num_objects":1255,"num_object_clones":619,"num_object_copies":2510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":105734,"num_read_kb":395151,"num_write":101029,"num_write_kb":403924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"88'103248","reported_seq":205136,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:26.772413+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:01:26.772413+0000","last_peered":"2023-10-03T18:01:26.772413+0000","last_clean":"2023-10-03T18:01:26.772413+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:01:26.772413+0000","last_undegraded":"2023-10-03T18:01:26.772413+0000","last_fullsized":"2023-10-03T18:01:26.772413+0000","mapping_epoch":12,"log_start":"84'93205","ondisk_log_start":"84'93205","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:54:42.067523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2633191756,"num_objects":1247,"num_object_clones":617,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103224,"num_read_kb":391314,"num_write":99950,"num_write_kb":399668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21864729038,"num_objects":10327,"num_object_clones":5120,"num_object_copies":20654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":857560,"num_read_kb":3250842,"num_write":831099,"num_write_kb":3323505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6804480000,"data_stored":12563190684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80454,"ondisk_log_size":80454,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738714,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2467172,"kb_used_data":1276188,"kb_used_omap":1,"kb_used_meta":1190974,"kb_avail":91904668,"statfs":{"total":96636764160,"available":94110380032,"internally_reserved":0,"allocated":1306816512,"data_stored":2411685052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1219557830},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,9,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.9689999999999999,"15min":2.673},"min":{"1min":0.219,"5min":0.187,"15min":0.16600000000000001},"max":{"1min":10.962999999999999,"5min":18.648,"15min":22.562000000000001},"last":2.2970000000000002},{"interface":"front","average":{"1min":4.2439999999999998,"5min":3.222,"15min":2.738},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.472,"5min":18.041,"15min":21.827999999999999},"last":7.6269999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:00:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1440000000000001,"5min":2.8719999999999999,"15min":2.4740000000000002},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":11.007,"5min":26.074000000000002,"15min":26.074000000000002},"last":7.516},{"interface":"front","average":{"1min":2.1960000000000002,"5min":2.0590000000000002,"15min":1.946},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.188},"max":{"1min":16.457000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":1.746}]}]},{"osd":0,"up_from":8,"seq":34359738713,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2630096,"kb_used_data":2046344,"kb_used_omap":347,"kb_used_meta":583396,"kb_avail":91741744,"statfs":{"total":96636764160,"available":93943545856,"internally_reserved":0,"allocated":2095456256,"data_stored":3869096444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":597397680},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":3.766,"15min":3.4700000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.143000000000001,"5min":21.928999999999998,"15min":21.928999999999998},"last":1.714},{"interface":"front","average":{"1min":3.012,"5min":3.02,"15min":3.6309999999999998},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.19},"max":{"1min":11.896000000000001,"5min":18.492000000000001,"15min":24.215},"last":1.675}]},{"osd":2,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.2879999999999998,"15min":2.4399999999999999},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.9880000000000004,"5min":12.256,"15min":12.256},"last":2.1259999999999999},{"interface":"front","average":{"1min":2.4260000000000002,"5min":2.3029999999999999,"15min":2.3050000000000002},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":11.94,"5min":12.630000000000001,"15min":13.677},"last":0.26800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738709,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4577572,"kb_used_data":3323740,"kb_used_omap":15,"kb_used_meta":1253808,"kb_avail":89794268,"statfs":{"total":96636764160,"available":91949330432,"internally_reserved":0,"allocated":3403509760,"data_stored":6282440019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1283900233},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,6,0,14,7,8,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.4050000000000002,"5min":3.4780000000000002,"15min":3.8130000000000002},"min":{"1min":0.33200000000000002,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":12.513999999999999,"5min":14.651,"15min":39.990000000000002},"last":4.79},{"interface":"front","average":{"1min":6.2430000000000003,"5min":3.9820000000000002,"15min":3.8380000000000001},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":20.716999999999999,"5min":24.481000000000002,"15min":37.381},"last":4.8230000000000004}]},{"osd":2,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.9199999999999999,"5min":3.1669999999999998,"15min":2.3740000000000001},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.17299999999999999},"max":{"1min":24.579000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":0.21199999999999999},{"interface":"front","average":{"1min":1.873,"5min":2.464,"15min":3.48},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.6029999999999998,"5min":18.677,"15min":22.739000000000001},"last":3.2549999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2094903296,"data_stored":3868968567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3403059200,"data_stored":6282414542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1306517504,"data_stored":2411807575,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:01:29.846 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-03T18:01:30.604 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:30.604 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":88,"fsid":"2a0c5350-120a-4bb6-a8cc-bda4017ada08","created":"2023-10-03T17:32:45.158778+0000","modified":"2023-10-03T18:00:28.434945+0000","last_up_change":"2023-10-03T17:32:54.879734+0000","last_in_change":"2023-10-03T17:32:46.195093+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T17:32:55.331164+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-10-03T17:33:00.866205+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":46,"snap_epoch":80,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"96941377-c8f6-45de-8108-5751af5af033","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6801","nonce":3351865178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6803","nonce":3351865178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6807","nonce":3351865178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3351865178},{"type":"v1","addr":"172.21.15.31:6805","nonce":3351865178}]},"public_addr":"172.21.15.31:6801/3351865178","cluster_addr":"172.21.15.31:6803/3351865178","heartbeat_back_addr":"172.21.15.31:6807/3351865178","heartbeat_front_addr":"172.21.15.31:6805/3351865178","state":["exists","up"]},{"osd":1,"uuid":"2895e424-6c96-426c-802d-3b91e5cafcb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6809","nonce":875549188}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6811","nonce":875549188}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6815","nonce":875549188}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":875549188},{"type":"v1","addr":"172.21.15.31:6813","nonce":875549188}]},"public_addr":"172.21.15.31:6809/875549188","cluster_addr":"172.21.15.31:6811/875549188","heartbeat_back_addr":"172.21.15.31:6815/875549188","heartbeat_front_addr":"172.21.15.31:6813/875549188","state":["exists","up"]},{"osd":2,"uuid":"58cf0e6f-2dfe-4737-8e92-0b5fc5950baf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6817","nonce":2591775043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6819","nonce":2591775043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6823","nonce":2591775043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":2591775043},{"type":"v1","addr":"172.21.15.31:6821","nonce":2591775043}]},"public_addr":"172.21.15.31:6817/2591775043","cluster_addr":"172.21.15.31:6819/2591775043","heartbeat_back_addr":"172.21.15.31:6823/2591775043","heartbeat_front_addr":"172.21.15.31:6821/2591775043","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.335718+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.378460+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T17:32:51.390043+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/874826947":"2023-10-03T19:00:26.497724+0000","172.21.15.31:0/2548654810":"2023-10-03T18:58:06.008423+0000","172.21.15.31:0/1481624570":"2023-10-03T18:55:44.327843+0000","172.21.15.31:0/1789969421":"2023-10-03T18:54:44.990183+0000","172.21.15.31:0/902319174":"2023-10-03T18:53:35.049953+0000","172.21.15.31:0/3173308107":"2023-10-03T18:52:24.887148+0000","172.21.15.31:0/3781251104":"2023-10-03T18:39:35.179001+0000","172.21.15.31:0/3463023637":"2023-10-03T18:37:23.512465+0000","172.21.15.31:0/1130397553":"2023-10-03T18:41:56.477965+0000","172.21.15.31:0/939425469":"2023-10-03T18:45:43.634556+0000","172.21.15.31:0/2220906593":"2023-10-03T18:35:24.901740+0000","172.21.15.31:0/3260346930":"2023-10-03T18:59:16.092142+0000","172.21.15.31:0/876723378":"2023-10-03T18:43:18.971912+0000","172.21.15.31:0/1609851556":"2023-10-03T18:40:46.154409+0000","172.21.15.31:0/2304558628":"2023-10-03T18:44:29.847025+0000","172.21.15.31:0/311399113":"2023-10-03T18:57:06.514036+0000","172.21.15.31:0/2110891374":"2023-10-03T18:36:13.001421+0000","172.21.15.31:0/2909175":"2023-10-03T18:45:29.527198+0000","172.21.15.31:0/87597209":"2023-10-03T18:38:23.964983+0000","172.21.15.31:0/3594551136":"2023-10-03T18:46:53.933086+0000","172.21.15.31:0/166292732":"2023-10-03T18:47:53.871211+0000","172.21.15.31:0/2507430896":"2023-10-03T18:51:25.080076+0000","172.21.15.31:0/3379124484":"2023-10-03T18:49:15.076803+0000","172.21.15.31:0/1378966707":"2023-10-03T18:50:14.781327+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T18:01:31.662 INFO:tasks.ceph:Scrubbing osd.0 2023-10-03T18:01:31.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-03T18:01:32.000 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-03T18:01:32.001 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T18:01:32.001 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-03T18:01:32.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-03T18:01:32.699 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 0 to deep-scrub 2023-10-03T18:01:32.721 INFO:tasks.ceph:Scrubbing osd.1 2023-10-03T18:01:32.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-03T18:01:33.044 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-03T18:01:33.045 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T18:01:33.045 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-03T18:01:33.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-03T18:01:33.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.386+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:01:33.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.448+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:01:33.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.448+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:01:33.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.448+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:01:33.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.454+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:01:33.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.454+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:01:33.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:33.454+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:01:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 1 to deep-scrub 2023-10-03T18:01:33.919 INFO:tasks.ceph:Scrubbing osd.2 2023-10-03T18:01:33.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-03T18:01:34.285 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-03T18:01:34.285 INFO:teuthology.orchestra.run.smithi031.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-03T18:01:34.285 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-03T18:01:34.310 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-03T18:01:35.084 INFO:teuthology.orchestra.run.smithi031.stderr:instructed osd(s) 2 to deep-scrub 2023-10-03T18:01:35.109 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:01:35.823 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:35.856 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:01:35.872 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":944,"stamp":"2023-10-03T18:01:34.404990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21875956718,"num_objects":10329,"num_object_clones":5120,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":860730,"num_read_kb":3261857,"num_write":833564,"num_write_kb":3334711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80397,"ondisk_log_size":80397,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9701372,"kb_used_data":6664484,"kb_used_omap":364,"kb_used_meta":3036499,"kb_avail":273414148,"statfs":{"total":289910292480,"available":279976087552,"internally_reserved":0,"allocated":6824431616,"data_stored":12581870603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3109375423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,7,3,11,21,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24375296,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7127,"num_read_kb":25177,"num_write":5358,"num_write_kb":21432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.014999"},"pg_stats":[{"pgid":"2.7","version":"88'107144","reported_seq":212364,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:32.032646+0000","last_change":"2023-10-03T17:34:48.146004+0000","last_active":"2023-10-03T18:01:32.032646+0000","last_peered":"2023-10-03T18:01:32.032646+0000","last_clean":"2023-10-03T18:01:32.032646+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:01:32.032646+0000","last_undegraded":"2023-10-03T18:01:32.032646+0000","last_fullsized":"2023-10-03T18:01:32.032646+0000","mapping_epoch":12,"log_start":"84'97108","ondisk_log_start":"84'97108","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:28:23.665197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2726928667,"num_objects":1285,"num_object_clones":638,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":106570,"num_read_kb":407354,"num_write":103842,"num_write_kb":415305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'97696","reported_seq":195186,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:32.032571+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:01:32.032571+0000","last_peered":"2023-10-03T18:01:32.032571+0000","last_clean":"2023-10-03T18:01:32.032571+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:01:32.032571+0000","last_undegraded":"2023-10-03T18:01:32.032571+0000","last_fullsized":"2023-10-03T18:01:32.032571+0000","mapping_epoch":12,"log_start":"84'87604","ondisk_log_start":"84'87604","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:03:34.306073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2518151386,"num_objects":1185,"num_object_clones":589,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":99694,"num_read_kb":376385,"num_write":95771,"num_write_kb":382910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'118407","reported_seq":232553,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:32.005657+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:01:32.005657+0000","last_peered":"2023-10-03T18:01:32.005657+0000","last_clean":"2023-10-03T18:01:32.005657+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:01:32.005657+0000","last_undegraded":"2023-10-03T18:01:32.005657+0000","last_fullsized":"2023-10-03T18:01:32.005657+0000","mapping_epoch":12,"log_start":"84'108405","ondisk_log_start":"84'108405","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:44:27.523925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2952081759,"num_objects":1395,"num_object_clones":691,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":116383,"num_read_kb":439022,"num_write":112412,"num_write_kb":449537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"88'111183","reported_seq":220914,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:33.207904+0000","last_change":"2023-10-03T17:34:48.152806+0000","last_active":"2023-10-03T18:01:33.207904+0000","last_peered":"2023-10-03T18:01:33.207904+0000","last_clean":"2023-10-03T18:01:33.207904+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:01:33.207904+0000","last_undegraded":"2023-10-03T18:01:33.207904+0000","last_fullsized":"2023-10-03T18:01:33.207904+0000","mapping_epoch":12,"log_start":"84'101106","ondisk_log_start":"84'101106","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:50:32.001473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2853929161,"num_objects":1344,"num_object_clones":668,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":111120,"num_read_kb":424646,"num_write":109259,"num_write_kb":436976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":229,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:00:28.459283+0000","last_change":"2023-10-03T17:32:57.192995+0000","last_active":"2023-10-03T18:00:28.459283+0000","last_peered":"2023-10-03T18:00:28.459283+0000","last_clean":"2023-10-03T18:00:28.459283+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:00:28.459283+0000","last_undegraded":"2023-10-03T18:00:28.459283+0000","last_fullsized":"2023-10-03T18:00:28.459283+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:32:56.183115+0000","last_clean_scrub_stamp":"2023-10-03T17:32:56.183115+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:10.715874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'113522","reported_seq":225451,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:31.925114+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:01:31.925114+0000","last_peered":"2023-10-03T18:01:31.925114+0000","last_clean":"2023-10-03T18:01:31.925114+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:01:31.925114+0000","last_undegraded":"2023-10-03T18:01:31.925114+0000","last_fullsized":"2023-10-03T18:01:31.925114+0000","mapping_epoch":12,"log_start":"84'103506","ondisk_log_start":"84'103506","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:32:01.804277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2900578689,"num_objects":1369,"num_object_clones":679,"num_object_copies":2738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":113601,"num_read_kb":433356,"num_write":110167,"num_write_kb":440590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'102918","reported_seq":204621,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:33.205574+0000","last_change":"2023-10-03T17:34:48.152786+0000","last_active":"2023-10-03T18:01:33.205574+0000","last_peered":"2023-10-03T18:01:33.205574+0000","last_clean":"2023-10-03T18:01:33.205574+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:01:33.205574+0000","last_undegraded":"2023-10-03T18:01:33.205574+0000","last_fullsized":"2023-10-03T18:01:33.205574+0000","mapping_epoch":12,"log_start":"84'92906","ondisk_log_start":"84'92906","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:38:12.166372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2643239181,"num_objects":1247,"num_object_clones":619,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103435,"num_read_kb":391902,"num_write":100425,"num_write_kb":401619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"88'103108","reported_seq":206917,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:29.345697+0000","last_change":"2023-10-03T17:34:48.140895+0000","last_active":"2023-10-03T18:01:29.345697+0000","last_peered":"2023-10-03T18:01:29.345697+0000","last_clean":"2023-10-03T18:01:29.345697+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:01:29.345697+0000","last_undegraded":"2023-10-03T18:01:29.345697+0000","last_fullsized":"2023-10-03T18:01:29.345697+0000","mapping_epoch":12,"log_start":"84'93105","ondisk_log_start":"84'93105","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:56:30.943255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2645705175,"num_objects":1255,"num_object_clones":619,"num_object_copies":2510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":106244,"num_read_kb":396556,"num_write":101282,"num_write_kb":404936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"88'103567","reported_seq":205776,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:31.988645+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:01:31.988645+0000","last_peered":"2023-10-03T18:01:31.988645+0000","last_clean":"2023-10-03T18:01:31.988645+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:01:31.988645+0000","last_undegraded":"2023-10-03T18:01:31.988645+0000","last_fullsized":"2023-10-03T18:01:31.988645+0000","mapping_epoch":12,"log_start":"84'93505","ondisk_log_start":"84'93505","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:54:42.067523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2634752332,"num_objects":1247,"num_object_clones":617,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":103577,"num_read_kb":392545,"num_write":100269,"num_write_kb":400944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21875366350,"num_objects":10327,"num_object_clones":5120,"num_object_copies":20654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":860624,"num_read_kb":3261766,"num_write":833427,"num_write_kb":3332817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6823616512,"data_stored":12582327196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80300,"ondisk_log_size":80300,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738715,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2474880,"kb_used_data":1279736,"kb_used_omap":1,"kb_used_meta":1195134,"kb_avail":91896960,"statfs":{"total":96636764160,"available":94102487040,"internally_reserved":0,"allocated":1310449664,"data_stored":2415318204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1223817670},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,5,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":2.319,"5min":2.9689999999999999,"15min":2.673},"min":{"1min":0.219,"5min":0.187,"15min":0.16600000000000001},"max":{"1min":10.962999999999999,"5min":18.648,"15min":22.562000000000001},"last":6.883},{"interface":"front","average":{"1min":4.2439999999999998,"5min":3.222,"15min":2.738},"min":{"1min":0.21199999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":12.472,"5min":18.041,"15min":21.827999999999999},"last":0.58599999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:00:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1440000000000001,"5min":2.8719999999999999,"15min":2.4740000000000002},"min":{"1min":0.22700000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":11.007,"5min":26.074000000000002,"15min":26.074000000000002},"last":0.26400000000000001},{"interface":"front","average":{"1min":2.1960000000000002,"5min":2.0590000000000002,"15min":1.946},"min":{"1min":0.23400000000000001,"5min":0.224,"15min":0.188},"max":{"1min":16.457000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.38200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738714,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2639700,"kb_used_data":2051788,"kb_used_omap":347,"kb_used_meta":587556,"kb_avail":91732140,"statfs":{"total":96636764160,"available":93933711360,"internally_reserved":0,"allocated":2101030912,"data_stored":3874671100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":601657520},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":3.766,"15min":3.4700000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.143000000000001,"5min":21.928999999999998,"15min":21.928999999999998},"last":4.0510000000000002},{"interface":"front","average":{"1min":3.012,"5min":3.02,"15min":3.6309999999999998},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.19},"max":{"1min":11.896000000000001,"5min":18.492000000000001,"15min":24.215},"last":3.98}]},{"osd":2,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.2879999999999998,"15min":2.4399999999999999},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.9880000000000004,"5min":12.256,"15min":12.256},"last":0.216},{"interface":"front","average":{"1min":2.4260000000000002,"5min":2.3029999999999999,"15min":2.3050000000000002},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":11.94,"5min":12.630000000000001,"15min":13.677},"last":5.1669999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738710,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4586792,"kb_used_data":3332960,"kb_used_omap":15,"kb_used_meta":1253808,"kb_avail":89785048,"statfs":{"total":96636764160,"available":91939889152,"internally_reserved":0,"allocated":3412951040,"data_stored":6291881299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1283900233},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,4,13,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.4050000000000002,"5min":3.4780000000000002,"15min":3.8130000000000002},"min":{"1min":0.33200000000000002,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":12.513999999999999,"5min":14.651,"15min":39.990000000000002},"last":2.742},{"interface":"front","average":{"1min":6.2430000000000003,"5min":3.9820000000000002,"15min":3.8380000000000001},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":20.716999999999999,"5min":24.481000000000002,"15min":37.381},"last":2.2090000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:00:41 2023","interfaces":[{"interface":"back","average":{"1min":4.9199999999999999,"5min":3.1669999999999998,"15min":2.3740000000000001},"min":{"1min":0.23499999999999999,"5min":0.218,"15min":0.17299999999999999},"max":{"1min":24.579000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":1.302},{"interface":"front","average":{"1min":1.873,"5min":2.464,"15min":3.48},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.6029999999999998,"5min":18.677,"15min":22.739000000000001},"last":10.654999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2100387840,"data_stored":3874453111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3413061632,"data_stored":6292416974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1310167040,"data_stored":2415457111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:01:35.875 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.876 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.876 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.876 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.876 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-03T17:32:56.183115+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=32, tm_sec=56, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.876 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.877 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.877 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.877 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:35.877 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:01:36.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:36.735+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:01:36.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:36.735+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:01:36.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:36.792+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:01:40.625 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:01:40.618+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:01:40.625 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:01:40.618+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:01:40.626 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:01:40.618+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:01:40.626 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:01:40.618+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:01:47.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:47.281+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:01:47.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:47.298+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:01:47.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:47.298+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:01:53.061 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:53.059+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:01:53.062 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:53.059+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:01:53.069 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:53.069+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:01:53.095 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:53.091+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:01:53.117 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:01:53.112+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:01:54.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:54.713+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:01:54.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:54.713+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:01:54.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:01:54.714+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:01:55.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:01:56.594 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:01:56.601 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:01:56.622 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2023-10-03T18:01:56.419364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21918092270,"num_objects":10329,"num_object_clones":5120,"num_object_copies":20658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10329,"num_whiteouts":0,"num_read":871963,"num_read_kb":3306309,"num_write":843194,"num_write_kb":3373231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80582,"ondisk_log_size":80582,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":9874144,"kb_used_data":6745608,"kb_used_omap":364,"kb_used_meta":3128147,"kb_avail":273241376,"statfs":{"total":289910292480,"available":279799169024,"internally_reserved":0,"allocated":6907502592,"data_stored":12664941579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3203222975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,4,10,12,16,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21803008,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5755,"num_read_kb":22744,"num_write":4976,"num_write_kb":19904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007994"},"pg_stats":[{"pgid":"2.7","version":"88'108374","reported_seq":214974,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:01:52.730089+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:01:52.730089+0000","last_peered":"2023-10-03T18:01:52.730089+0000","last_clean":"2023-10-03T18:01:52.730089+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:01:52.730089+0000","last_undegraded":"2023-10-03T18:01:52.730089+0000","last_fullsized":"2023-10-03T18:01:52.730089+0000","mapping_epoch":12,"log_start":"84'98308","ondisk_log_start":"84'98308","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":56,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 15s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2731823387,"num_objects":1285,"num_object_clones":638,"num_object_copies":2570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1285,"num_whiteouts":0,"num_read":107929,"num_read_kb":412726,"num_write":105005,"num_write_kb":419957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"88'98805","reported_seq":197591,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:52.709614+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:01:52.709614+0000","last_peered":"2023-10-03T18:01:52.709614+0000","last_clean":"2023-10-03T18:01:52.709614+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:01:52.709614+0000","last_undegraded":"2023-10-03T18:01:52.709614+0000","last_fullsized":"2023-10-03T18:01:52.709614+0000","mapping_epoch":12,"log_start":"84'88804","ondisk_log_start":"84'88804","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2522915034,"num_objects":1185,"num_object_clones":589,"num_object_copies":2370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1185,"num_whiteouts":0,"num_read":100981,"num_read_kb":381477,"num_write":96861,"num_write_kb":387270,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"88'119757","reported_seq":235370,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:52.708763+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:01:52.708763+0000","last_peered":"2023-10-03T18:01:52.708763+0000","last_clean":"2023-10-03T18:01:52.708763+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:01:52.708763+0000","last_undegraded":"2023-10-03T18:01:52.708763+0000","last_fullsized":"2023-10-03T18:01:52.708763+0000","mapping_epoch":12,"log_start":"86'109705","ondisk_log_start":"86'109705","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2957848927,"num_objects":1395,"num_object_clones":691,"num_object_copies":2790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":117829,"num_read_kb":444742,"num_write":113686,"num_write_kb":454633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"88'112385","reported_seq":223543,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:53.377300+0000","last_change":"2023-10-03T18:01:50.768307+0000","last_active":"2023-10-03T18:01:53.377300+0000","last_peered":"2023-10-03T18:01:53.377300+0000","last_clean":"2023-10-03T18:01:53.377300+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:01:53.377300+0000","last_undegraded":"2023-10-03T18:01:53.377300+0000","last_fullsized":"2023-10-03T18:01:53.377300+0000","mapping_epoch":12,"log_start":"84'102306","ondisk_log_start":"84'102306","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2859155657,"num_objects":1344,"num_object_clones":668,"num_object_copies":2688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":112530,"num_read_kb":430246,"num_write":110453,"num_write_kb":441752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":240,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:34.463450+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:01:34.463450+0000","last_peered":"2023-10-03T18:01:34.463450+0000","last_clean":"2023-10-03T18:01:34.463450+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:01:34.463450+0000","last_undegraded":"2023-10-03T18:01:34.463450+0000","last_fullsized":"2023-10-03T18:01:34.463450+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"88'114788","reported_seq":228193,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:52.711017+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:01:52.711017+0000","last_peered":"2023-10-03T18:01:52.711017+0000","last_clean":"2023-10-03T18:01:52.711017+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:01:52.711017+0000","last_undegraded":"2023-10-03T18:01:52.711017+0000","last_fullsized":"2023-10-03T18:01:52.711017+0000","mapping_epoch":12,"log_start":"86'104706","ondisk_log_start":"86'104706","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2905891201,"num_objects":1369,"num_object_clones":679,"num_object_copies":2738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":115067,"num_read_kb":439172,"num_write":111395,"num_write_kb":445502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"88'104099","reported_seq":207179,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:01:53.391656+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:01:53.391656+0000","last_peered":"2023-10-03T18:01:53.391656+0000","last_clean":"2023-10-03T18:01:53.391656+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:01:53.391656+0000","last_undegraded":"2023-10-03T18:01:53.391656+0000","last_fullsized":"2023-10-03T18:01:53.391656+0000","mapping_epoch":12,"log_start":"84'94006","ondisk_log_start":"84'94006","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":48,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2648142093,"num_objects":1247,"num_object_clones":619,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":104788,"num_read_kb":397250,"num_write":101569,"num_write_kb":406195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"88'104555","reported_seq":210022,"reported_epoch":88,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:01:54.610619+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:01:54.610619+0000","last_peered":"2023-10-03T18:01:54.610619+0000","last_clean":"2023-10-03T18:01:54.610619+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:01:54.610619+0000","last_undegraded":"2023-10-03T18:01:54.610619+0000","last_fullsized":"2023-10-03T18:01:54.610619+0000","mapping_epoch":12,"log_start":"84'94505","ondisk_log_start":"84'94505","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":84,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 21s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2651697623,"num_objects":1255,"num_object_clones":619,"num_object_copies":2510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":107864,"num_read_kb":402972,"num_write":102659,"num_write_kb":410444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"88'104765","reported_seq":208275,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-10-03T18:01:52.728935+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:01:52.728935+0000","last_peered":"2023-10-03T18:01:52.728935+0000","last_clean":"2023-10-03T18:01:52.728935+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:01:52.728935+0000","last_undegraded":"2023-10-03T18:01:52.728935+0000","last_fullsized":"2023-10-03T18:01:52.728935+0000","mapping_epoch":12,"log_start":"84'94705","ondisk_log_start":"84'94705","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2640027980,"num_objects":1247,"num_object_clones":617,"num_object_copies":2494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1247,"num_whiteouts":0,"num_read":104869,"num_read_kb":397633,"num_write":101429,"num_write_kb":405584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21917501902,"num_objects":10327,"num_object_clones":5120,"num_object_copies":20654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10327,"num_whiteouts":0,"num_read":871857,"num_read_kb":3306218,"num_write":843057,"num_write_kb":3371337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6905573376,"data_stored":12664284060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80483,"ondisk_log_size":80483,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738719,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2498724,"kb_used_data":1293852,"kb_used_omap":1,"kb_used_meta":1204862,"kb_avail":91873116,"statfs":{"total":96636764160,"available":94078070784,"internally_reserved":0,"allocated":1324904448,"data_stored":2429772988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1233779142},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":2.8690000000000002,"15min":2.718},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":8.6950000000000003,"5min":12.539999999999999,"15min":22.562000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":3.3010000000000002,"5min":3.2530000000000001,"15min":2.766},"min":{"1min":0.22500000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":14.52,"5min":18.041,"15min":21.827999999999999},"last":6.6859999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:01:44 2023","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.71,"15min":2.496},"min":{"1min":0.26400000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":8.548,"5min":26.074000000000002,"15min":26.074000000000002},"last":1.375},{"interface":"front","average":{"1min":3.2309999999999999,"5min":2.181,"15min":2.0099999999999998},"min":{"1min":0.26700000000000002,"5min":0.224,"15min":0.188},"max":{"1min":8.1999999999999993,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738719,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2699940,"kb_used_data":2080092,"kb_used_omap":347,"kb_used_meta":619492,"kb_avail":91671900,"statfs":{"total":96636764160,"available":93872025600,"internally_reserved":0,"allocated":2130014208,"data_stored":3903654396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":634359984},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,4,9,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":3.133,"5min":3.766,"15min":3.4700000000000002},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.143000000000001,"5min":21.928999999999998,"15min":21.928999999999998},"last":0.42999999999999999},{"interface":"front","average":{"1min":3.012,"5min":3.02,"15min":3.6309999999999998},"min":{"1min":0.219,"5min":0.19400000000000001,"15min":0.19},"max":{"1min":11.896000000000001,"5min":18.492000000000001,"15min":24.215},"last":0.39800000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:00:49 2023","interfaces":[{"interface":"back","average":{"1min":1.98,"5min":2.2879999999999998,"15min":2.4399999999999999},"min":{"1min":0.26000000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":5.9880000000000004,"5min":12.256,"15min":12.256},"last":1.5920000000000001},{"interface":"front","average":{"1min":2.4260000000000002,"5min":2.3029999999999999,"15min":2.3050000000000002},"min":{"1min":0.23599999999999999,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":11.94,"5min":12.630000000000001,"15min":13.677},"last":0.36299999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738714,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4675480,"kb_used_data":3371664,"kb_used_omap":15,"kb_used_meta":1303792,"kb_avail":89696360,"statfs":{"total":96636764160,"available":91849072640,"internally_reserved":0,"allocated":3452583936,"data_stored":6331514195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1335083849},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,0,4,6,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":3.1070000000000002,"5min":3.359,"15min":3.7719999999999998},"min":{"1min":0.17100000000000001,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":10.537000000000001,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":3.8839999999999999,"5min":4.3230000000000004,"15min":3.9620000000000002},"min":{"1min":0.28899999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.891999999999999,"5min":24.481000000000002,"15min":37.381},"last":3.7650000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":4.5949999999999998,"5min":3.859,"15min":2.4809999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.338000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":3.8660000000000001},{"interface":"front","average":{"1min":2.472,"5min":2.2730000000000001,"15min":3.3410000000000002},"min":{"1min":0.22500000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.654999999999999,"5min":10.991,"15min":22.739000000000001},"last":3.0640000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2129485824,"data_stored":3903551095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3451412480,"data_stored":6330767822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1324675072,"data_stored":2429965143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:01:56.624 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.625 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.625 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.625 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.626 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.626 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.626 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.627 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:01:56.627 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:02:04.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:04.886+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:04.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:04.886+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:04.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:04.948+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:05.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:04.999+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:05.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:05.022+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:06.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:06.916+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:02:06.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:06.970+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:06.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:06.970+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:06.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:06.970+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:07.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:06.997+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:07.070 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.069+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:07.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.147+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:07.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.147+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:07.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.169+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:07.203 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.202+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:07.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.220+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:07.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.925+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:07.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.930+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:07.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.958+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:07.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:07.985+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:08.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.036+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:08.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.036+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:08.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.123+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:08.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.123+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:08.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.134+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:08.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.844+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:02:08.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.844+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:08.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.844+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:08.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.847+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:08.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.885+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:08.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.885+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:08.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.886+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:08.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.886+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:08.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.917+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:08.979 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.977+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:08.979 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:08.977+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:13.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.689+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:13.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.694+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:13.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.724+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:13.744 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.743+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:13.745 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.743+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:13.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.769+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:13.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.769+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:13.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:13.769+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:14.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.690+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:02:14.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.703+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:14.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.703+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:14.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.715+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:14.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:14.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.806+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:14.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.849+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:14.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.849+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:14.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:14.929+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:15.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.039+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:15.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.039+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:15.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.694+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:15.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.698+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:15.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.703+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:15.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.709+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:15.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.716+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:15.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.717+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:15.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:15.727+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:16.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:02:17.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.156+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:17.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.226+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:17.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.226+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:17.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.258+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:17.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.258+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:17.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.339+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:17.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.339+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:17.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.339+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:02:17.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.339+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:17.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.339+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:17.449 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:02:17.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.452+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:17.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.452+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:17.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.452+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:17.466 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:02:17.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:17.470+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:17.479 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":968,"stamp":"2023-10-03T18:02:16.434713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21949538273,"num_objects":12930,"num_object_clones":7701,"num_object_copies":25860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12930,"num_whiteouts":0,"num_read":882184,"num_read_kb":3340679,"num_write":850750,"num_write_kb":3403215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80462,"ondisk_log_size":80462,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10148456,"kb_used_data":6805388,"kb_used_omap":364,"kb_used_meta":3342675,"kb_avail":272967064,"statfs":{"total":289910292480,"available":279518273536,"internally_reserved":0,"allocated":6968717312,"data_stored":16173716095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3422899647},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,0,3,10,7,3,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13860914,"num_objects":2218,"num_object_clones":2217,"num_object_copies":4436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2218,"num_whiteouts":0,"num_read":3999,"num_read_kb":14574,"num_write":3368,"num_write_kb":13433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005936"},"pg_stats":[{"pgid":"2.7","version":"91'109752","reported_seq":217149,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:14.070505+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:02:14.070505+0000","last_peered":"2023-10-03T18:02:14.070505+0000","last_clean":"2023-10-03T18:02:14.070505+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:02:14.070505+0000","last_undegraded":"2023-10-03T18:02:14.070505+0000","last_fullsized":"2023-10-03T18:02:14.070505+0000","mapping_epoch":12,"log_start":"86'99710","ondisk_log_start":"86'99710","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":162,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 36s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2735726926,"num_objects":1620,"num_object_clones":972,"num_object_copies":3240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":109148,"num_read_kb":416998,"num_write":105965,"num_write_kb":423776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'100009","reported_seq":199678,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:14.026878+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:02:14.026878+0000","last_peered":"2023-10-03T18:02:14.026878+0000","last_clean":"2023-10-03T18:02:14.026878+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:02:14.026878+0000","last_undegraded":"2023-10-03T18:02:14.026878+0000","last_fullsized":"2023-10-03T18:02:14.026878+0000","mapping_epoch":12,"log_start":"86'90008","ondisk_log_start":"86'90008","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2526831062,"num_objects":1469,"num_object_clones":868,"num_object_copies":2938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1469,"num_whiteouts":0,"num_read":102209,"num_read_kb":385779,"num_write":97772,"num_write_kb":390890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'121210","reported_seq":237776,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:13.982122+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:02:13.982122+0000","last_peered":"2023-10-03T18:02:13.982122+0000","last_clean":"2023-10-03T18:02:13.982122+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:02:13.982122+0000","last_undegraded":"2023-10-03T18:02:13.982122+0000","last_fullsized":"2023-10-03T18:02:13.982122+0000","mapping_epoch":12,"log_start":"86'111109","ondisk_log_start":"86'111109","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2962358623,"num_objects":1734,"num_object_clones":1030,"num_object_copies":3468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1734,"num_whiteouts":0,"num_read":119343,"num_read_kb":449622,"num_write":114748,"num_write_kb":458845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"91'113671","reported_seq":225583,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:13.417329+0000","last_change":"2023-10-03T18:02:13.016755+0000","last_active":"2023-10-03T18:02:13.417329+0000","last_peered":"2023-10-03T18:02:13.417329+0000","last_clean":"2023-10-03T18:02:13.417329+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:02:13.417329+0000","last_undegraded":"2023-10-03T18:02:13.417329+0000","last_fullsized":"2023-10-03T18:02:13.417329+0000","mapping_epoch":12,"log_start":"86'103610","ondisk_log_start":"86'103610","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2863133176,"num_objects":1679,"num_object_clones":997,"num_object_copies":3358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1679,"num_whiteouts":0,"num_read":113710,"num_read_kb":434378,"num_write":111395,"num_write_kb":445484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":243,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T18:02:01.519265+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:02:01.519265+0000","last_peered":"2023-10-03T18:02:01.519265+0000","last_clean":"2023-10-03T18:02:01.519265+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:02:01.519265+0000","last_undegraded":"2023-10-03T18:02:01.519265+0000","last_fullsized":"2023-10-03T18:02:01.519265+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'116225","reported_seq":230419,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:13.959781+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:02:13.959781+0000","last_peered":"2023-10-03T18:02:13.959781+0000","last_clean":"2023-10-03T18:02:13.959781+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:02:13.959781+0000","last_undegraded":"2023-10-03T18:02:13.959781+0000","last_fullsized":"2023-10-03T18:02:13.959781+0000","mapping_epoch":12,"log_start":"86'106208","ondisk_log_start":"86'106208","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2910138905,"num_objects":1738,"num_object_clones":1045,"num_object_copies":3476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1738,"num_whiteouts":0,"num_read":116347,"num_read_kb":443650,"num_write":112438,"num_write_kb":449647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'105409","reported_seq":209202,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:13.424688+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:02:13.424688+0000","last_peered":"2023-10-03T18:02:13.424688+0000","last_clean":"2023-10-03T18:02:13.424688+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:02:13.424688+0000","last_undegraded":"2023-10-03T18:02:13.424688+0000","last_fullsized":"2023-10-03T18:02:13.424688+0000","mapping_epoch":12,"log_start":"86'95310","ondisk_log_start":"86'95310","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":131,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 34s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2651984242,"num_objects":1571,"num_object_clones":941,"num_object_copies":3142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1571,"num_whiteouts":0,"num_read":105986,"num_read_kb":401190,"num_write":102494,"num_write_kb":409862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"91'105819","reported_seq":212067,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:14.906801+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:02:14.906801+0000","last_peered":"2023-10-03T18:02:14.906801+0000","last_clean":"2023-10-03T18:02:14.906801+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:02:14.906801+0000","last_undegraded":"2023-10-03T18:02:14.906801+0000","last_fullsized":"2023-10-03T18:02:14.906801+0000","mapping_epoch":12,"log_start":"86'95809","ondisk_log_start":"86'95809","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":222,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 42s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2655216137,"num_objects":1583,"num_object_clones":946,"num_object_copies":3166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1583,"num_whiteouts":0,"num_read":109197,"num_read_kb":406990,"num_write":103510,"num_write_kb":413809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"91'105940","reported_seq":210329,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:13.913917+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:02:13.913917+0000","last_peered":"2023-10-03T18:02:13.913917+0000","last_clean":"2023-10-03T18:02:13.913917+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:02:13.913917+0000","last_undegraded":"2023-10-03T18:02:13.913917+0000","last_fullsized":"2023-10-03T18:02:13.913917+0000","mapping_epoch":12,"log_start":"86'95908","ondisk_log_start":"86'95908","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2643558834,"num_objects":1534,"num_object_clones":902,"num_object_copies":3068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1534,"num_whiteouts":0,"num_read":106138,"num_read_kb":401981,"num_write":102291,"num_write_kb":409008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21948947905,"num_objects":12928,"num_object_clones":7701,"num_object_copies":25856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12928,"num_whiteouts":0,"num_read":882078,"num_read_kb":3340588,"num_write":850613,"num_write_kb":3401321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6966456320,"data_stored":16268684162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80363,"ondisk_log_size":80363,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738723,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2554644,"kb_used_data":1305420,"kb_used_omap":1,"kb_used_meta":1249214,"kb_avail":91817196,"statfs":{"total":96636764160,"available":94020808704,"internally_reserved":0,"allocated":1336750080,"data_stored":3099438018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1279195590},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,2,1,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":2.8690000000000002,"15min":2.718},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":8.6950000000000003,"5min":12.539999999999999,"15min":22.562000000000001},"last":3.0249999999999999},{"interface":"front","average":{"1min":3.3010000000000002,"5min":3.2530000000000001,"15min":2.766},"min":{"1min":0.22500000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":14.52,"5min":18.041,"15min":21.827999999999999},"last":5.4909999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:01:44 2023","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.71,"15min":2.496},"min":{"1min":0.26400000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":8.548,"5min":26.074000000000002,"15min":26.074000000000002},"last":1.135},{"interface":"front","average":{"1min":3.2309999999999999,"5min":2.181,"15min":2.0099999999999998},"min":{"1min":0.26700000000000002,"5min":0.224,"15min":0.188},"max":{"1min":8.1999999999999993,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.20699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738723,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2748712,"kb_used_data":2097504,"kb_used_omap":347,"kb_used_meta":650852,"kb_avail":91623128,"statfs":{"total":96636764160,"available":93822083072,"internally_reserved":0,"allocated":2147844096,"data_stored":4971253917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":666472624},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":4.0640000000000001,"15min":3.6669999999999998},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":15.602,"5min":15.602,"15min":21.928999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":2.9060000000000001,"5min":2.6989999999999998,"15min":3.6320000000000001},"min":{"1min":0.25700000000000001,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":11.981999999999999,"5min":18.109000000000002,"15min":24.215},"last":0.41999999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.1190000000000002,"15min":2.3879999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":11.291,"5min":12.256,"15min":12.256},"last":0.96399999999999997},{"interface":"front","average":{"1min":2.4249999999999998,"5min":2.3069999999999999,"15min":2.3519999999999999},"min":{"1min":0.214,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":7.8890000000000002,"5min":12.630000000000001,"15min":13.677},"last":0.73999999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738718,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4845100,"kb_used_data":3402464,"kb_used_omap":15,"kb_used_meta":1442608,"kb_avail":89526740,"statfs":{"total":96636764160,"available":91675381760,"internally_reserved":0,"allocated":3484123136,"data_stored":8103024160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1477231433},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,5,6,2,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":3.1070000000000002,"5min":3.359,"15min":3.7719999999999998},"min":{"1min":0.17100000000000001,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":10.537000000000001,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.311},{"interface":"front","average":{"1min":3.8839999999999999,"5min":4.3230000000000004,"15min":3.9620000000000002},"min":{"1min":0.28899999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.891999999999999,"5min":24.481000000000002,"15min":37.381},"last":12.067}]},{"osd":2,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":4.5949999999999998,"5min":3.859,"15min":2.4809999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.338000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":11.496},{"interface":"front","average":{"1min":2.472,"5min":2.2730000000000001,"15min":3.3410000000000002},"min":{"1min":0.22500000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.654999999999999,"5min":10.991,"15min":22.739000000000001},"last":12.255000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2147192832,"data_stored":5026150462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3483004928,"data_stored":8119025089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1336258560,"data_stored":3123508611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:02:17.481 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.481 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.481 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.481 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.482 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.482 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.482 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.482 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:17.482 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:02:18.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:18.719+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:18.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:18.735+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:18.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:18.980+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:18.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:18.980+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:19.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.003+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:19.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.003+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:19.074 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.073+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:19.074 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.073+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:19.087 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.085+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:19.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:19.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:19.174+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:20.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:20.589+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:20.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:20.597+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:21.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.877+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:21.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.879+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:21.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.910+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:21.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.917+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:21.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.919+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:21.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.945+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:21.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.945+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:21.965 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.965+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:21.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.980+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:21.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:21.980+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:22.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.061+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:22.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.193+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:22.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.193+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:22.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.193+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:22.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:22.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:22.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:22.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:22.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:22.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:22.197+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:22.876 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:02:22.876+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:02:22.877 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:02:22.876+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:02:24.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.039+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:24.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.039+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:24.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.062+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:24.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.062+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:24.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.065+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:24.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.104+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:24.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:24.125+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:28.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:28.589+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:28.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:28.589+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:28.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:28.624+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:28.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:28.632+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:28.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:28.632+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:34.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:34.682+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:34.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:34.720+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:34.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:34.773+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:34.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:34.777+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:37.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:37.068+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:37.484 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:02:38.309 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:02:38.336 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:02:38.350 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":978,"stamp":"2023-10-03T18:02:36.448238+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":21980905441,"num_objects":14962,"num_object_clones":9733,"num_object_copies":29924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14962,"num_whiteouts":0,"num_read":891035,"num_read_kb":3372084,"num_write":858350,"num_write_kb":3433615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80586,"ondisk_log_size":80586,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10039748,"kb_used_data":6866412,"kb_used_omap":364,"kb_used_meta":3172947,"kb_avail":273075772,"statfs":{"total":289910292480,"available":279629590528,"internally_reserved":0,"allocated":7031205888,"data_stored":19016315519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3249098175},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,5,3,8,11,5,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17723392,"num_objects":807,"num_object_clones":807,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":807,"num_whiteouts":0,"num_read":5393,"num_read_kb":17741,"num_write":4292,"num_write_kb":17168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008816"},"pg_stats":[{"pgid":"2.7","version":"91'111083","reported_seq":219264,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:34.847128+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:02:34.847128+0000","last_peered":"2023-10-03T18:02:34.847128+0000","last_clean":"2023-10-03T18:02:34.847128+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:02:34.847128+0000","last_undegraded":"2023-10-03T18:02:34.847128+0000","last_fullsized":"2023-10-03T18:02:34.847128+0000","mapping_epoch":12,"log_start":"86'101011","ondisk_log_start":"86'101011","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":251,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 57s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2739827022,"num_objects":1866,"num_object_clones":1218,"num_object_copies":3732,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1866,"num_whiteouts":0,"num_read":110237,"num_read_kb":420992,"num_write":106958,"num_write_kb":427748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'101107","reported_seq":201450,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:34.822231+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:02:34.822231+0000","last_peered":"2023-10-03T18:02:34.822231+0000","last_clean":"2023-10-03T18:02:34.822231+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:02:34.822231+0000","last_undegraded":"2023-10-03T18:02:34.822231+0000","last_fullsized":"2023-10-03T18:02:34.822231+0000","mapping_epoch":12,"log_start":"86'91012","ondisk_log_start":"86'91012","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2530267606,"num_objects":1711,"num_object_clones":1110,"num_object_copies":3422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":103208,"num_read_kb":389309,"num_write":98607,"num_write_kb":394230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'122635","reported_seq":240063,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:34.815636+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:02:34.815636+0000","last_peered":"2023-10-03T18:02:34.815636+0000","last_clean":"2023-10-03T18:02:34.815636+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:02:34.815636+0000","last_undegraded":"2023-10-03T18:02:34.815636+0000","last_fullsized":"2023-10-03T18:02:34.815636+0000","mapping_epoch":12,"log_start":"86'112613","ondisk_log_start":"86'112613","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2966819167,"num_objects":2009,"num_object_clones":1305,"num_object_copies":4018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2009,"num_whiteouts":0,"num_read":120617,"num_read_kb":454014,"num_write":115818,"num_write_kb":463125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"91'114881","reported_seq":227513,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:33.553558+0000","last_change":"2023-10-03T18:02:32.202983+0000","last_active":"2023-10-03T18:02:33.553558+0000","last_peered":"2023-10-03T18:02:33.553558+0000","last_clean":"2023-10-03T18:02:33.553558+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:02:33.553558+0000","last_undegraded":"2023-10-03T18:02:33.553558+0000","last_fullsized":"2023-10-03T18:02:33.553558+0000","mapping_epoch":12,"log_start":"86'104812","ondisk_log_start":"86'104812","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2866987512,"num_objects":1948,"num_object_clones":1266,"num_object_copies":3896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1948,"num_whiteouts":0,"num_read":114761,"num_read_kb":438212,"num_write":112330,"num_write_kb":449224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":243,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T18:02:01.519265+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:02:01.519265+0000","last_peered":"2023-10-03T18:02:01.519265+0000","last_clean":"2023-10-03T18:02:01.519265+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:02:01.519265+0000","last_undegraded":"2023-10-03T18:02:01.519265+0000","last_fullsized":"2023-10-03T18:02:01.519265+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'117511","reported_seq":232510,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:34.783283+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:02:34.783283+0000","last_peered":"2023-10-03T18:02:34.783283+0000","last_clean":"2023-10-03T18:02:34.783283+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:02:34.783283+0000","last_undegraded":"2023-10-03T18:02:34.783283+0000","last_fullsized":"2023-10-03T18:02:34.783283+0000","mapping_epoch":12,"log_start":"86'107414","ondisk_log_start":"86'107414","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2914222617,"num_objects":1992,"num_object_clones":1299,"num_object_copies":3984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1992,"num_whiteouts":0,"num_read":117487,"num_read_kb":447842,"num_write":113427,"num_write_kb":453603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'106590","reported_seq":211159,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:33.573340+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:02:33.573340+0000","last_peered":"2023-10-03T18:02:33.573340+0000","last_clean":"2023-10-03T18:02:33.573340+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:02:33.573340+0000","last_undegraded":"2023-10-03T18:02:33.573340+0000","last_fullsized":"2023-10-03T18:02:33.573340+0000","mapping_epoch":12,"log_start":"86'96511","ondisk_log_start":"86'96511","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":238,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 54s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2655613298,"num_objects":1803,"num_object_clones":1173,"num_object_copies":3606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1803,"num_whiteouts":0,"num_read":107081,"num_read_kb":405037,"num_write":103377,"num_write_kb":413394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"91'107053","reported_seq":214098,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:35.098142+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:02:35.098142+0000","last_peered":"2023-10-03T18:02:35.098142+0000","last_clean":"2023-10-03T18:02:35.098142+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:02:35.098142+0000","last_undegraded":"2023-10-03T18:02:35.098142+0000","last_fullsized":"2023-10-03T18:02:35.098142+0000","mapping_epoch":12,"log_start":"86'97013","ondisk_log_start":"86'97013","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":317,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 62s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2659041801,"num_objects":1823,"num_object_clones":1186,"num_object_copies":3646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1823,"num_whiteouts":0,"num_read":110374,"num_read_kb":410931,"num_write":104438,"num_write_kb":417521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"91'107224","reported_seq":212293,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:34.817861+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:02:34.817861+0000","last_peered":"2023-10-03T18:02:34.817861+0000","last_clean":"2023-10-03T18:02:34.817861+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:02:34.817861+0000","last_undegraded":"2023-10-03T18:02:34.817861+0000","last_fullsized":"2023-10-03T18:02:34.817861+0000","mapping_epoch":12,"log_start":"86'97211","ondisk_log_start":"86'97211","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2647536050,"num_objects":1808,"num_object_clones":1176,"num_object_copies":3616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1808,"num_whiteouts":0,"num_read":107164,"num_read_kb":405656,"num_write":103258,"num_write_kb":412876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":21980315073,"num_objects":14960,"num_object_clones":9733,"num_object_copies":29920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14960,"num_whiteouts":0,"num_read":890929,"num_read_kb":3371993,"num_write":858213,"num_write_kb":3431721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7028482048,"data_stored":19032570754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80487,"ondisk_log_size":80487,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738727,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2604176,"kb_used_data":1316360,"kb_used_omap":1,"kb_used_meta":1287806,"kb_avail":91767664,"statfs":{"total":96636764160,"available":93970087936,"internally_reserved":0,"allocated":1347952640,"data_stored":3631045570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1318713798},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:42 2023","interfaces":[{"interface":"back","average":{"1min":2.2040000000000002,"5min":2.8690000000000002,"15min":2.718},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":8.6950000000000003,"5min":12.539999999999999,"15min":22.562000000000001},"last":2.0990000000000002},{"interface":"front","average":{"1min":3.3010000000000002,"5min":3.2530000000000001,"15min":2.766},"min":{"1min":0.22500000000000001,"5min":0.193,"15min":0.17999999999999999},"max":{"1min":14.52,"5min":18.041,"15min":21.827999999999999},"last":6.1289999999999996}]},{"osd":1,"last update":"Tue Oct 3 18:01:44 2023","interfaces":[{"interface":"back","average":{"1min":2.5009999999999999,"5min":2.71,"15min":2.496},"min":{"1min":0.26400000000000001,"5min":0.19600000000000001,"15min":0.18099999999999999},"max":{"1min":8.548,"5min":26.074000000000002,"15min":26.074000000000002},"last":6.1619999999999999},{"interface":"front","average":{"1min":3.2309999999999999,"5min":2.181,"15min":2.0099999999999998},"min":{"1min":0.26700000000000002,"5min":0.224,"15min":0.188},"max":{"1min":8.1999999999999993,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.28699999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738727,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2448048,"kb_used_data":2116904,"kb_used_omap":347,"kb_used_meta":330788,"kb_avail":91923792,"statfs":{"total":96636764160,"available":94129963008,"internally_reserved":0,"allocated":2167709696,"data_stored":5869568157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":338727088},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,1,0,1,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":4.0640000000000001,"15min":3.6669999999999998},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":15.602,"5min":15.602,"15min":21.928999999999998},"last":0.624},{"interface":"front","average":{"1min":2.9060000000000001,"5min":2.6989999999999998,"15min":3.6320000000000001},"min":{"1min":0.25700000000000001,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":11.981999999999999,"5min":18.109000000000002,"15min":24.215},"last":0.28799999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.1190000000000002,"15min":2.3879999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":11.291,"5min":12.256,"15min":12.256},"last":1.8400000000000001},{"interface":"front","average":{"1min":2.4249999999999998,"5min":2.3069999999999999,"15min":2.3519999999999999},"min":{"1min":0.214,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":7.8890000000000002,"5min":12.630000000000001,"15min":13.677},"last":0.32300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738722,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4987524,"kb_used_data":3433148,"kb_used_omap":15,"kb_used_meta":1554352,"kb_avail":89384316,"statfs":{"total":96636764160,"available":91529539584,"internally_reserved":0,"allocated":3515543552,"data_stored":9515701792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1591657289},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,5,8,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":3.1070000000000002,"5min":3.359,"15min":3.7719999999999998},"min":{"1min":0.17100000000000001,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":10.537000000000001,"5min":12.513999999999999,"15min":39.990000000000002},"last":2.4649999999999999},{"interface":"front","average":{"1min":3.8839999999999999,"5min":4.3230000000000004,"15min":3.9620000000000002},"min":{"1min":0.28899999999999998,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":10.891999999999999,"5min":24.481000000000002,"15min":37.381},"last":1.673}]},{"osd":2,"last update":"Tue Oct 3 18:01:41 2023","interfaces":[{"interface":"back","average":{"1min":4.5949999999999998,"5min":3.859,"15min":2.4809999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.338000000000001,"5min":24.579000000000001,"15min":24.579000000000001},"last":2.4289999999999998},{"interface":"front","average":{"1min":2.472,"5min":2.2730000000000001,"15min":3.3410000000000002},"min":{"1min":0.22500000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":10.654999999999999,"5min":10.991,"15min":22.739000000000001},"last":0.34100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2166411264,"data_stored":5869320254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3514523648,"data_stored":9521241537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1347547136,"data_stored":3642008963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:02:38.352 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.352 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.352 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.352 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.353 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.353 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.353 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.353 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:38.353 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:02:42.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.866+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:42.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.881+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:42.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.881+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:42.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.882+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:42.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.912+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:42.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:42.926+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:44.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.236+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:44.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.236+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:44.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.270+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:44.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.290+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:44.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.290+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:44.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.883+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:44.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.883+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:44.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.884+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:44.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:44.951+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:45.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.009+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:45.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.239+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:02:45.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.239+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:02:45.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.239+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:02:45.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.239+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:02:45.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.265+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:02:45.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.312+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:45.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.312+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:45.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.312+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:45.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.315+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:45.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.315+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:45.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:45.315+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:48.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:48.729+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:48.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:48.729+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:48.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:48.755+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:48.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:48.799+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:48.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:48.802+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:54.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:54.079+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:54.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:54.079+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:54.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:54.182+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:54.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:54.236+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:56.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.196+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:02:56.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.229+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:02:56.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.229+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:02:56.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.235+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:02:56.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.235+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:02:56.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:02:56.243+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:02:57.265 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:02:57.265+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:02:57.266 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:02:57.265+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:02:57.266 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:02:57.265+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:02:58.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:02:59.283 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:02:59.322 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:02:59.344 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":989,"stamp":"2023-10-03T18:02:58.456032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22017900513,"num_objects":15402,"num_object_clones":10173,"num_object_copies":30804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15402,"num_whiteouts":0,"num_read":899761,"num_read_kb":3406652,"num_write":867321,"num_write_kb":3469499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80602,"ondisk_log_size":80602,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10232028,"kb_used_data":6937028,"kb_used_omap":364,"kb_used_meta":3294611,"kb_avail":272883492,"statfs":{"total":289910292480,"available":279432695808,"internally_reserved":0,"allocated":7103516672,"data_stored":19692667519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3373682111},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,2,3,12,8,24,17,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19234816,"num_objects":115,"num_object_clones":115,"num_object_copies":230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":115,"num_whiteouts":0,"num_read":4453,"num_read_kb":17660,"num_write":4654,"num_write_kb":18616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006141"},"pg_stats":[{"pgid":"2.7","version":"91'112374","reported_seq":221605,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:55.793234+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:02:55.793234+0000","last_peered":"2023-10-03T18:02:55.793234+0000","last_clean":"2023-10-03T18:02:55.793234+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:02:55.793234+0000","last_undegraded":"2023-10-03T18:02:55.793234+0000","last_fullsized":"2023-10-03T18:02:55.793234+0000","mapping_epoch":12,"log_start":"86'102311","ondisk_log_start":"86'102311","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":328,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 78s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2744553806,"num_objects":1918,"num_object_clones":1270,"num_object_copies":3836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1918,"num_whiteouts":0,"num_read":111323,"num_read_kb":425316,"num_write":108106,"num_write_kb":432340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"91'102270","reported_seq":203589,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:55.764120+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:02:55.764120+0000","last_peered":"2023-10-03T18:02:55.764120+0000","last_clean":"2023-10-03T18:02:55.764120+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:02:55.764120+0000","last_undegraded":"2023-10-03T18:02:55.764120+0000","last_fullsized":"2023-10-03T18:02:55.764120+0000","mapping_epoch":12,"log_start":"86'92214","ondisk_log_start":"86'92214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2534707670,"num_objects":1771,"num_object_clones":1170,"num_object_copies":3542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1771,"num_whiteouts":0,"num_read":104242,"num_read_kb":393417,"num_write":99690,"num_write_kb":398562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"91'123943","reported_seq":242532,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:55.707886+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:02:55.707886+0000","last_peered":"2023-10-03T18:02:55.707886+0000","last_clean":"2023-10-03T18:02:55.707886+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:02:55.707886+0000","last_undegraded":"2023-10-03T18:02:55.707886+0000","last_fullsized":"2023-10-03T18:02:55.707886+0000","mapping_epoch":12,"log_start":"87'113914","ondisk_log_start":"87'113914","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2971611487,"num_objects":2073,"num_object_clones":1369,"num_object_copies":4146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2073,"num_whiteouts":0,"num_read":121826,"num_read_kb":458798,"num_write":116987,"num_write_kb":467801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"91'116091","reported_seq":229830,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:53.662044+0000","last_change":"2023-10-03T18:02:53.078070+0000","last_active":"2023-10-03T18:02:53.662044+0000","last_peered":"2023-10-03T18:02:53.662044+0000","last_clean":"2023-10-03T18:02:53.662044+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:02:53.662044+0000","last_undegraded":"2023-10-03T18:02:53.662044+0000","last_fullsized":"2023-10-03T18:02:53.662044+0000","mapping_epoch":12,"log_start":"86'106012","ondisk_log_start":"86'106012","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2871693816,"num_objects":2010,"num_object_clones":1328,"num_object_copies":4020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":115913,"num_read_kb":442772,"num_write":113472,"num_write_kb":453792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":243,"reported_epoch":90,"state":"active+clean","last_fresh":"2023-10-03T18:02:01.519265+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:02:01.519265+0000","last_peered":"2023-10-03T18:02:01.519265+0000","last_clean":"2023-10-03T18:02:01.519265+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:02:01.519265+0000","last_undegraded":"2023-10-03T18:02:01.519265+0000","last_fullsized":"2023-10-03T18:02:01.519265+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"91'118867","reported_seq":234964,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:55.726144+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:02:55.726144+0000","last_peered":"2023-10-03T18:02:55.726144+0000","last_clean":"2023-10-03T18:02:55.726144+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:02:55.726144+0000","last_undegraded":"2023-10-03T18:02:55.726144+0000","last_fullsized":"2023-10-03T18:02:55.726144+0000","mapping_epoch":12,"log_start":"86'108814","ondisk_log_start":"86'108814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2919436825,"num_objects":2044,"num_object_clones":1351,"num_object_copies":4088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2044,"num_whiteouts":0,"num_read":118626,"num_read_kb":452338,"num_write":114692,"num_write_kb":458663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"91'107788","reported_seq":213384,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:53.652212+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:02:53.652212+0000","last_peered":"2023-10-03T18:02:53.652212+0000","last_clean":"2023-10-03T18:02:53.652212+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:02:53.652212+0000","last_undegraded":"2023-10-03T18:02:53.652212+0000","last_fullsized":"2023-10-03T18:02:53.652212+0000","mapping_epoch":12,"log_start":"86'97712","ondisk_log_start":"86'97712","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":307,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 75s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2660122994,"num_objects":1860,"num_object_clones":1230,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1860,"num_whiteouts":0,"num_read":108148,"num_read_kb":409273,"num_write":104463,"num_write_kb":417738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"91'108198","reported_seq":216211,"reported_epoch":91,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:02:55.351523+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:02:55.351523+0000","last_peered":"2023-10-03T18:02:55.351523+0000","last_clean":"2023-10-03T18:02:55.351523+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:02:55.351523+0000","last_undegraded":"2023-10-03T18:02:55.351523+0000","last_fullsized":"2023-10-03T18:02:55.351523+0000","mapping_epoch":12,"log_start":"86'98114","ondisk_log_start":"86'98114","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":428,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 82s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2663268873,"num_objects":1865,"num_object_clones":1228,"num_object_copies":3730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1865,"num_whiteouts":0,"num_read":111355,"num_read_kb":414811,"num_write":105457,"num_write_kb":421597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"91'108374","reported_seq":214458,"reported_epoch":91,"state":"active+clean","last_fresh":"2023-10-03T18:02:55.709586+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:02:55.709586+0000","last_peered":"2023-10-03T18:02:55.709586+0000","last_clean":"2023-10-03T18:02:55.709586+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:02:55.709586+0000","last_undegraded":"2023-10-03T18:02:55.709586+0000","last_fullsized":"2023-10-03T18:02:55.709586+0000","mapping_epoch":12,"log_start":"86'98311","ondisk_log_start":"86'98311","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2651914674,"num_objects":1859,"num_object_clones":1227,"num_object_copies":3718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1859,"num_whiteouts":0,"num_read":108222,"num_read_kb":409836,"num_write":104317,"num_write_kb":417112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22017310145,"num_objects":15400,"num_object_clones":10173,"num_object_copies":30800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15400,"num_whiteouts":0,"num_read":899655,"num_read_kb":3406561,"num_write":867184,"num_write_kb":3467605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7101804544,"data_stored":19696466818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80503,"ondisk_log_size":80503,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738731,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2644144,"kb_used_data":1329960,"kb_used_omap":1,"kb_used_meta":1314174,"kb_avail":91727696,"statfs":{"total":96636764160,"available":93929160704,"internally_reserved":0,"allocated":1361879040,"data_stored":3777436610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1345714630},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":2.4089999999999998,"15min":2.6059999999999999},"min":{"1min":0.182,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":3.6659999999999999,"5min":12.539999999999999,"15min":22.562000000000001},"last":3.806},{"interface":"front","average":{"1min":3.2050000000000001,"5min":3.2719999999999998,"15min":2.8380000000000001},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":11.457000000000001,"5min":18.041,"15min":21.827999999999999},"last":7.3200000000000003}]},{"osd":1,"last update":"Tue Oct 3 18:02:45 2023","interfaces":[{"interface":"back","average":{"1min":2.4409999999999998,"5min":2.762,"15min":2.5600000000000001},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.9329999999999998,"5min":26.074000000000002,"15min":26.074000000000002},"last":0.316},{"interface":"front","average":{"1min":2.3860000000000001,"5min":2.3039999999999998,"15min":2.0670000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":10.206,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.22}]}]},{"osd":0,"up_from":8,"seq":34359738731,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2482404,"kb_used_data":2137372,"kb_used_omap":347,"kb_used_meta":344676,"kb_avail":91889436,"statfs":{"total":96636764160,"available":94094782464,"internally_reserved":0,"allocated":2188668928,"data_stored":6063382685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":352948400},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,2,10,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":4.9889999999999999,"5min":4.0640000000000001,"15min":3.6669999999999998},"min":{"1min":0.17299999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":15.602,"5min":15.602,"15min":21.928999999999998},"last":6.8760000000000003},{"interface":"front","average":{"1min":2.9060000000000001,"5min":2.6989999999999998,"15min":3.6320000000000001},"min":{"1min":0.25700000000000001,"5min":0.20399999999999999,"15min":0.19400000000000001},"max":{"1min":11.981999999999999,"5min":18.109000000000002,"15min":24.215},"last":6.774}]},{"osd":2,"last update":"Tue Oct 3 18:01:55 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.1190000000000002,"15min":2.3879999999999999},"min":{"1min":0.216,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":11.291,"5min":12.256,"15min":12.256},"last":6.4820000000000002},{"interface":"front","average":{"1min":2.4249999999999998,"5min":2.3069999999999999,"15min":2.3519999999999999},"min":{"1min":0.214,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":7.8890000000000002,"5min":12.630000000000001,"15min":13.677},"last":4.891}]}]},{"osd":1,"up_from":8,"seq":34359738726,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5105480,"kb_used_data":3469696,"kb_used_omap":15,"kb_used_meta":1635760,"kb_avail":89266360,"statfs":{"total":96636764160,"available":91408752640,"internally_reserved":0,"allocated":3552968704,"data_stored":9851848224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1675019081},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,6,5,9,11,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":3.4279999999999999,"15min":3.6899999999999999},"min":{"1min":0.19,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":11.653,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":3.109,"5min":4.3300000000000001,"15min":3.827},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":12.067,"5min":24.481000000000002,"15min":37.381},"last":9.4480000000000004}]},{"osd":2,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":3.7370000000000001,"15min":2.5150000000000001},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":11.496,"5min":24.579000000000001,"15min":24.579000000000001},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.9950000000000001,"15min":3.2770000000000001},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":12.255000000000001,"5min":12.255000000000001,"15min":22.739000000000001},"last":3.0990000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2188296192,"data_stored":6065616958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3551940608,"data_stored":9851256257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1361567744,"data_stored":3779593603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:02:59.345 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.345 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.346 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.346 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.346 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.346 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.347 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.347 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:02:59.347 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:03:01.701 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:01.700+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:01.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:01.744+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:02.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.082+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:03:02.084 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.084+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:02.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.148+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:02.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.149+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:02.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.176+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:02.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.176+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:02.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.183+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:02.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.222+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:02.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:02.222+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:05.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:05.925+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:05.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:05.990+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:05.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:05.990+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:05.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:05.991+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:07.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:07.226+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:07.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:07.226+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:07.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:07.226+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:07.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:07.227+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:07.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:07.227+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:08.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:08.567+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:08.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:08.629+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:13.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.565+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:13.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.597+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:13.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.597+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:13.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.610+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:13.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.610+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:13.647 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.645+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:13.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.774+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:03:13.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.836+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:03:13.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.896+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:13.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.931+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:13.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:13.931+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:14.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:14.007+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:14.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:14.007+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:14.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:14.023+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:14.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:14.036+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:14.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:14.059+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:15.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.563+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:15.604 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:03:15.603+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:03:15.605 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:03:15.603+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:03:15.605 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:03:15.603+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:03:15.605 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:03:15.603+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:03:15.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.630+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:15.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.648+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:15.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.648+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:15.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.648+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:15.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.649+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:15.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.658+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:15.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:15.658+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:17.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.044+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:17.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.114+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:17.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.114+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:17.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.116+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:17.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.125+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:17.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:17.145+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:19.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:03:20.009 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:03:20.028 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:03:20.055 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1001,"stamp":"2023-10-03T18:03:18.466656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22049108948,"num_objects":18354,"num_object_clones":13105,"num_object_copies":36708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18354,"num_whiteouts":0,"num_read":909139,"num_read_kb":3436701,"num_write":874802,"num_write_kb":3499183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80476,"ondisk_log_size":80476,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10568948,"kb_used_data":6995416,"kb_used_omap":364,"kb_used_meta":3573139,"kb_avail":272546572,"statfs":{"total":289910292480,"available":279087689728,"internally_reserved":0,"allocated":7163305984,"data_stored":23656776729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":373313,"internal_metadata":3658894783},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,37,6,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19046757,"num_objects":2704,"num_object_clones":2697,"num_object_copies":5408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2704,"num_whiteouts":0,"num_read":5224,"num_read_kb":17892,"num_write":4501,"num_write_kb":17926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009310"},"pg_stats":[{"pgid":"2.7","version":"93'113821","reported_seq":223662,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:17.488932+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:03:17.488932+0000","last_peered":"2023-10-03T18:03:17.488932+0000","last_clean":"2023-10-03T18:03:17.488932+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:03:17.488932+0000","last_undegraded":"2023-10-03T18:03:17.488932+0000","last_fullsized":"2023-10-03T18:03:17.488932+0000","mapping_epoch":12,"log_start":"88'103814","ondisk_log_start":"88'103814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":394,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 100s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2748522880,"num_objects":2323,"num_object_clones":1674,"num_object_copies":4646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2323,"num_whiteouts":0,"num_read":112462,"num_read_kb":429152,"num_write":109069,"num_write_kb":436171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'103465","reported_seq":205352,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:17.376897+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:03:17.376897+0000","last_peered":"2023-10-03T18:03:17.376897+0000","last_clean":"2023-10-03T18:03:17.376897+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:03:17.376897+0000","last_undegraded":"2023-10-03T18:03:17.376897+0000","last_fullsized":"2023-10-03T18:03:17.376897+0000","mapping_epoch":12,"log_start":"86'93417","ondisk_log_start":"86'93417","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2538070587,"num_objects":2130,"num_object_clones":1527,"num_object_copies":4260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2130,"num_whiteouts":0,"num_read":105246,"num_read_kb":396770,"num_write":100509,"num_write_kb":401823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'125462","reported_seq":244730,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:17.304995+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:03:17.304995+0000","last_peered":"2023-10-03T18:03:17.304995+0000","last_clean":"2023-10-03T18:03:17.304995+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:03:17.304995+0000","last_undegraded":"2023-10-03T18:03:17.304995+0000","last_fullsized":"2023-10-03T18:03:17.304995+0000","mapping_epoch":12,"log_start":"88'115416","ondisk_log_start":"88'115416","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2975830468,"num_objects":2495,"num_object_clones":1789,"num_object_copies":4990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":123197,"num_read_kb":462950,"num_write":118013,"num_write_kb":471863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"93'117406","reported_seq":231801,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:13.730912+0000","last_change":"2023-10-03T18:03:13.222400+0000","last_active":"2023-10-03T18:03:13.730912+0000","last_peered":"2023-10-03T18:03:13.730912+0000","last_clean":"2023-10-03T18:03:13.730912+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:03:13.730912+0000","last_undegraded":"2023-10-03T18:03:13.730912+0000","last_fullsized":"2023-10-03T18:03:13.730912+0000","mapping_epoch":12,"log_start":"88'107313","ondisk_log_start":"88'107313","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2875724431,"num_objects":2350,"num_object_clones":1665,"num_object_copies":4700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":117035,"num_read_kb":446572,"num_write":114438,"num_write_kb":457629,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":245,"reported_epoch":92,"state":"active+clean","last_fresh":"2023-10-03T18:03:00.728186+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:03:00.728186+0000","last_peered":"2023-10-03T18:03:00.728186+0000","last_clean":"2023-10-03T18:03:00.728186+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:03:00.728186+0000","last_undegraded":"2023-10-03T18:03:00.728186+0000","last_fullsized":"2023-10-03T18:03:00.728186+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'120318","reported_seq":237047,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:17.327230+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:03:17.327230+0000","last_peered":"2023-10-03T18:03:17.327230+0000","last_clean":"2023-10-03T18:03:17.327230+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:03:17.327230+0000","last_undegraded":"2023-10-03T18:03:17.327230+0000","last_fullsized":"2023-10-03T18:03:17.327230+0000","mapping_epoch":12,"log_start":"88'110316","ondisk_log_start":"88'110316","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2923807410,"num_objects":2448,"num_object_clones":1752,"num_object_copies":4896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2448,"num_whiteouts":0,"num_read":119805,"num_read_kb":456325,"num_write":115705,"num_write_kb":462688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'109078","reported_seq":215309,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:13.731504+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:03:13.731504+0000","last_peered":"2023-10-03T18:03:13.731504+0000","last_clean":"2023-10-03T18:03:13.731504+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:03:13.731504+0000","last_undegraded":"2023-10-03T18:03:13.731504+0000","last_fullsized":"2023-10-03T18:03:13.731504+0000","mapping_epoch":12,"log_start":"86'99016","ondisk_log_start":"86'99016","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":443,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 95s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2663719383,"num_objects":2169,"num_object_clones":1537,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2169,"num_whiteouts":0,"num_read":109279,"num_read_kb":412787,"num_write":105338,"num_write_kb":421205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"93'109493","reported_seq":218198,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:15.604517+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:03:15.604517+0000","last_peered":"2023-10-03T18:03:15.604517+0000","last_clean":"2023-10-03T18:03:15.604517+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:03:15.604517+0000","last_undegraded":"2023-10-03T18:03:15.604517+0000","last_fullsized":"2023-10-03T18:03:15.604517+0000","mapping_epoch":12,"log_start":"86'99419","ondisk_log_start":"86'99419","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":565,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 102s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2666840736,"num_objects":2190,"num_object_clones":1550,"num_object_copies":4380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2190,"num_whiteouts":0,"num_read":112650,"num_read_kb":418466,"num_write":106309,"num_write_kb":424960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'109763","reported_seq":216465,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:17.310249+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:03:17.310249+0000","last_peered":"2023-10-03T18:03:17.310249+0000","last_clean":"2023-10-03T18:03:17.310249+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:03:17.310249+0000","last_undegraded":"2023-10-03T18:03:17.310249+0000","last_fullsized":"2023-10-03T18:03:17.310249+0000","mapping_epoch":12,"log_start":"87'99718","ondisk_log_start":"87'99718","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2656002685,"num_objects":2247,"num_object_clones":1611,"num_object_copies":4494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2247,"num_whiteouts":0,"num_read":109359,"num_read_kb":413588,"num_write":105284,"num_write_kb":420950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22048518580,"num_objects":18352,"num_object_clones":13105,"num_object_copies":36704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18352,"num_whiteouts":0,"num_read":909033,"num_read_kb":3436610,"num_write":874665,"num_write_kb":3497289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7162101760,"data_stored":23813825384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26884,"internal_metadata":0},"log_size":80377,"ondisk_log_size":80377,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738735,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2727216,"kb_used_data":1341352,"kb_used_omap":1,"kb_used_meta":1385854,"kb_avail":91644624,"statfs":{"total":96636764160,"available":93844094976,"internally_reserved":0,"allocated":1373544448,"data_stored":4446400659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1419114950},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,3,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":2.4089999999999998,"15min":2.6059999999999999},"min":{"1min":0.182,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":3.6659999999999999,"5min":12.539999999999999,"15min":22.562000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":3.2050000000000001,"5min":3.2719999999999998,"15min":2.8380000000000001},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":11.457000000000001,"5min":18.041,"15min":21.827999999999999},"last":0.23599999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:02:45 2023","interfaces":[{"interface":"back","average":{"1min":2.4409999999999998,"5min":2.762,"15min":2.5600000000000001},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.9329999999999998,"5min":26.074000000000002,"15min":26.074000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.3860000000000001,"5min":2.3039999999999998,"15min":2.0670000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":10.206,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738735,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2568128,"kb_used_data":2155064,"kb_used_omap":347,"kb_used_meta":412708,"kb_avail":91803712,"statfs":{"total":96636764160,"available":94007001088,"internally_reserved":0,"allocated":2206785536,"data_stored":7248807479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":422613168},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.657,"5min":3.8730000000000002,"15min":3.7120000000000002},"min":{"1min":0.26400000000000001,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.420999999999999,"5min":15.602,"15min":21.928999999999998},"last":5.335},{"interface":"front","average":{"1min":2.5649999999999999,"5min":2.4620000000000002,"15min":3.5659999999999998},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":10.092000000000001,"5min":11.981999999999999,"15min":24.215},"last":2.6480000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.944,"5min":2.1779999999999999,"15min":2.452},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":4.5750000000000002},{"interface":"front","average":{"1min":1.889,"5min":2.129,"15min":2.3490000000000002},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":5.681,"5min":12.630000000000001,"15min":13.677},"last":6.5789999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738730,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5273604,"kb_used_data":3499000,"kb_used_omap":15,"kb_used_meta":1774576,"kb_avail":89098236,"statfs":{"total":96636764160,"available":91236593664,"internally_reserved":0,"allocated":3582976000,"data_stored":11961568591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15543,"internal_metadata":1817166665},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,36,1,2,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":3.4279999999999999,"15min":3.6899999999999999},"min":{"1min":0.19,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":11.653,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":3.109,"5min":4.3300000000000001,"15min":3.827},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":12.067,"5min":24.481000000000002,"15min":37.381},"last":0.34699999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":3.7370000000000001,"15min":2.5150000000000001},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":11.496,"5min":24.579000000000001,"15min":24.579000000000001},"last":3.802},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.9950000000000001,"15min":3.2770000000000001},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":12.255000000000001,"5min":12.255000000000001,"15min":22.739000000000001},"last":0.317}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2206208000,"data_stored":7308462748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3582803968,"data_stored":12041081268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13949,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1373089792,"data_stored":4464281368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:03:20.056 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.057 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.058 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:20.058 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:03:20.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.119+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:20.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.147+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:20.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.147+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:20.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.194+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:20.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.196+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:20.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:20.203+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:21.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:21.098+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:21.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:21.100+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:21.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:21.110+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:21.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:21.110+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:21.139 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:21.137+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:28.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:03:28.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:28.693 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.684+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:28.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.718+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:28.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.739+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:28.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.739+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:28.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.753+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:28.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.753+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:28.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:28.754+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:33.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:33.978+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:33.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:33.978+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:33.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:33.978+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:33.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:33.978+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:35.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.233+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:03:35.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.233+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:03:35.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.269+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:35.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.269+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:35.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.275+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:35.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.296+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:35.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.296+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:35.303 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.304+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:35.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.315+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:35.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:35.315+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:36.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.660+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:36.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.674+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:36.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.674+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:36.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.716+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:36.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.716+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:36.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.744+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:36.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.760+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:03:36.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.767+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:03:36.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.767+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:03:36.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.768+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:03:36.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.768+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:36.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.786+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:36.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.786+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:36.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.786+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:36.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.793+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:36.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.793+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:36.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:36.799+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:37.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.370+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:03:37.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.443+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:37.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.443+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:37.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.443+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:37.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.443+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:37.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.470+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:37.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:37.470+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:40.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:03:40.724 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:03:40.739 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:03:40.754 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1012,"stamp":"2023-10-03T18:03:40.475602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22084408276,"num_objects":20146,"num_object_clones":14897,"num_object_copies":40292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20146,"num_whiteouts":0,"num_read":918860,"num_read_kb":3471225,"num_write":883080,"num_write_kb":3532295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80447,"ondisk_log_size":80447,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10461248,"kb_used_data":7060712,"kb_used_omap":368,"kb_used_meta":3400143,"kb_avail":272654272,"statfs":{"total":289910292480,"available":279197974528,"internally_reserved":0,"allocated":7230169088,"data_stored":26339152921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":377612,"internal_metadata":3481746676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,6,4,8,12,10,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19668992,"num_objects":680,"num_object_clones":680,"num_object_copies":1360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":0,"num_read":6045,"num_read_kb":19916,"num_write":4582,"num_write_kb":18328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002894"},"pg_stats":[{"pgid":"2.7","version":"93'115204","reported_seq":225870,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:38.346454+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:03:38.346454+0000","last_peered":"2023-10-03T18:03:38.346454+0000","last_clean":"2023-10-03T18:03:38.346454+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:03:38.346454+0000","last_undegraded":"2023-10-03T18:03:38.346454+0000","last_fullsized":"2023-10-03T18:03:38.346454+0000","mapping_epoch":12,"log_start":"88'105116","ondisk_log_start":"88'105116","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":529,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 121s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2753081728,"num_objects":2513,"num_object_clones":1864,"num_object_copies":5026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2513,"num_whiteouts":0,"num_read":113555,"num_read_kb":433130,"num_write":110130,"num_write_kb":440415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'104621","reported_seq":207293,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:38.333571+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:03:38.333571+0000","last_peered":"2023-10-03T18:03:38.333571+0000","last_clean":"2023-10-03T18:03:38.333571+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:03:38.333571+0000","last_undegraded":"2023-10-03T18:03:38.333571+0000","last_fullsized":"2023-10-03T18:03:38.333571+0000","mapping_epoch":12,"log_start":"88'94619","ondisk_log_start":"88'94619","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2542129723,"num_objects":2321,"num_object_clones":1718,"num_object_copies":4642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2321,"num_whiteouts":0,"num_read":106315,"num_read_kb":400575,"num_write":101455,"num_write_kb":405607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'126826","reported_seq":247034,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:38.344228+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:03:38.344228+0000","last_peered":"2023-10-03T18:03:38.344228+0000","last_clean":"2023-10-03T18:03:38.344228+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:03:38.344228+0000","last_undegraded":"2023-10-03T18:03:38.344228+0000","last_fullsized":"2023-10-03T18:03:38.344228+0000","mapping_epoch":12,"log_start":"88'116819","ondisk_log_start":"88'116819","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2980315588,"num_objects":2711,"num_object_clones":2005,"num_object_copies":5422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2711,"num_whiteouts":0,"num_read":124528,"num_read_kb":467522,"num_write":119085,"num_write_kb":476151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"93'118853","reported_seq":234241,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:38.840550+0000","last_change":"2023-10-03T18:03:27.302669+0000","last_active":"2023-10-03T18:03:38.840550+0000","last_peered":"2023-10-03T18:03:38.840550+0000","last_clean":"2023-10-03T18:03:38.840550+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:03:38.840550+0000","last_undegraded":"2023-10-03T18:03:38.840550+0000","last_fullsized":"2023-10-03T18:03:38.840550+0000","mapping_epoch":12,"log_start":"88'108815","ondisk_log_start":"88'108815","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2880770703,"num_objects":2624,"num_object_clones":1939,"num_object_copies":5248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2624,"num_whiteouts":0,"num_read":118379,"num_read_kb":451546,"num_write":115595,"num_write_kb":462257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":246,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:02.456735+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:03:02.456735+0000","last_peered":"2023-10-03T18:03:02.456735+0000","last_clean":"2023-10-03T18:03:02.456735+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:03:02.456735+0000","last_undegraded":"2023-10-03T18:03:02.456735+0000","last_fullsized":"2023-10-03T18:03:02.456735+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'121638","reported_seq":239242,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:38.304695+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:03:38.304695+0000","last_peered":"2023-10-03T18:03:38.304695+0000","last_clean":"2023-10-03T18:03:38.304695+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:03:38.304695+0000","last_undegraded":"2023-10-03T18:03:38.304695+0000","last_fullsized":"2023-10-03T18:03:38.304695+0000","mapping_epoch":12,"log_start":"88'111618","ondisk_log_start":"88'111618","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2928243378,"num_objects":2678,"num_object_clones":1982,"num_object_copies":5356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2678,"num_whiteouts":0,"num_read":121005,"num_read_kb":460698,"num_write":116757,"num_write_kb":466896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'110600","reported_seq":217761,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:38.840980+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:03:38.840980+0000","last_peered":"2023-10-03T18:03:38.840980+0000","last_clean":"2023-10-03T18:03:38.840980+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:03:38.840980+0000","last_undegraded":"2023-10-03T18:03:38.840980+0000","last_fullsized":"2023-10-03T18:03:38.840980+0000","mapping_epoch":12,"log_start":"88'100518","ondisk_log_start":"88'100518","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":579,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 120s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2668564951,"num_objects":2438,"num_object_clones":1806,"num_object_copies":4876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2438,"num_whiteouts":0,"num_read":110590,"num_read_kb":417448,"num_write":106474,"num_write_kb":425749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"93'110758","reported_seq":220346,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:35.879290+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:03:35.879290+0000","last_peered":"2023-10-03T18:03:35.879290+0000","last_clean":"2023-10-03T18:03:35.879290+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:03:35.879290+0000","last_undegraded":"2023-10-03T18:03:35.879290+0000","last_fullsized":"2023-10-03T18:03:35.879290+0000","mapping_epoch":12,"log_start":"88'100721","ondisk_log_start":"88'100721","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":659,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 123s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2670826144,"num_objects":2428,"num_object_clones":1788,"num_object_copies":4856,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2428,"num_whiteouts":0,"num_read":113937,"num_read_kb":422752,"num_write":107253,"num_write_kb":428736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'110895","reported_seq":218412,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:38.277119+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:03:38.277119+0000","last_peered":"2023-10-03T18:03:38.277119+0000","last_clean":"2023-10-03T18:03:38.277119+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:03:38.277119+0000","last_undegraded":"2023-10-03T18:03:38.277119+0000","last_fullsized":"2023-10-03T18:03:38.277119+0000","mapping_epoch":12,"log_start":"88'100821","ondisk_log_start":"88'100821","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2659885693,"num_objects":2431,"num_object_clones":1795,"num_object_copies":4862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2431,"num_whiteouts":0,"num_read":110445,"num_read_kb":417463,"num_write":106194,"num_write_kb":424590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22083817908,"num_objects":20144,"num_object_clones":14897,"num_object_copies":40288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20144,"num_whiteouts":0,"num_read":918754,"num_read_kb":3471134,"num_write":882943,"num_write_kb":3530401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7228538880,"data_stored":26368701288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31183,"internal_metadata":0},"log_size":80348,"ondisk_log_size":80348,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738740,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2799992,"kb_used_data":1355504,"kb_used_omap":1,"kb_used_meta":1444478,"kb_avail":91571848,"statfs":{"total":96636764160,"available":93769572352,"internally_reserved":0,"allocated":1388036096,"data_stored":5068820627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1479145926},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":2.4089999999999998,"15min":2.6059999999999999},"min":{"1min":0.182,"5min":0.17899999999999999,"15min":0.17799999999999999},"max":{"1min":3.6659999999999999,"5min":12.539999999999999,"15min":22.562000000000001},"last":9.9870000000000001},{"interface":"front","average":{"1min":3.2050000000000001,"5min":3.2719999999999998,"15min":2.8380000000000001},"min":{"1min":0.22700000000000001,"5min":0.21199999999999999,"15min":0.17999999999999999},"max":{"1min":11.457000000000001,"5min":18.041,"15min":21.827999999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Tue Oct 3 18:02:45 2023","interfaces":[{"interface":"back","average":{"1min":2.4409999999999998,"5min":2.762,"15min":2.5600000000000001},"min":{"1min":0.22600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":9.9329999999999998,"5min":26.074000000000002,"15min":26.074000000000002},"last":0.441},{"interface":"front","average":{"1min":2.3860000000000001,"5min":2.3039999999999998,"15min":2.0670000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":10.206,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738739,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2655900,"kb_used_data":2174804,"kb_used_omap":347,"kb_used_meta":480740,"kb_avail":91715940,"statfs":{"total":96636764160,"available":93917122560,"internally_reserved":0,"allocated":2226999296,"data_stored":8093677111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":492277936},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,4,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.657,"5min":3.8730000000000002,"15min":3.7120000000000002},"min":{"1min":0.26400000000000001,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.420999999999999,"5min":15.602,"15min":21.928999999999998},"last":30.852},{"interface":"front","average":{"1min":2.5649999999999999,"5min":2.4620000000000002,"15min":3.5659999999999998},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":10.092000000000001,"5min":11.981999999999999,"15min":24.215},"last":2.1850000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.944,"5min":2.1779999999999999,"15min":2.452},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":3.4079999999999999},{"interface":"front","average":{"1min":1.889,"5min":2.129,"15min":2.3490000000000002},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":5.681,"5min":12.630000000000001,"15min":13.677},"last":4.9850000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738734,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5005356,"kb_used_data":3530404,"kb_used_omap":19,"kb_used_meta":1474924,"kb_avail":89366484,"statfs":{"total":96636764160,"available":91511279616,"internally_reserved":0,"allocated":3615133696,"data_stored":13176655183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19842,"internal_metadata":1510322814},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,4,5,7,6,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.8380000000000001,"5min":3.4279999999999999,"15min":3.6899999999999999},"min":{"1min":0.19,"5min":0.16400000000000001,"15min":0.16300000000000001},"max":{"1min":11.653,"5min":12.513999999999999,"15min":39.990000000000002},"last":2.6230000000000002},{"interface":"front","average":{"1min":3.109,"5min":4.3300000000000001,"15min":3.827},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":12.067,"5min":24.481000000000002,"15min":37.381},"last":4.3460000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:02:44 2023","interfaces":[{"interface":"back","average":{"1min":2.1890000000000001,"5min":3.7370000000000001,"15min":2.5150000000000001},"min":{"1min":0.23100000000000001,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":11.496,"5min":24.579000000000001,"15min":24.579000000000001},"last":2.637},{"interface":"front","average":{"1min":2.1789999999999998,"5min":1.9950000000000001,"15min":3.2770000000000001},"min":{"1min":0.215,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":12.255000000000001,"5min":12.255000000000001,"15min":22.739000000000001},"last":4.3899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2225664000,"data_stored":8093489820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3615326208,"data_stored":13202366900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18248,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1387548672,"data_stored":5072844568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:03:40.756 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.756 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.756 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.756 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.757 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.757 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.757 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.758 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:03:40.758 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:03:41.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:41.426+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:41.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:41.432+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:42.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.638+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:42.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.669+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:42.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.669+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:42.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.842+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:42.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.878+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:42.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:42.878+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:43.892 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:43.891+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:43.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:43.915+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:43.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:43.923+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:54.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.513+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:03:54.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.520+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:03:54.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.520+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:03:54.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.538+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:03:54.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.538+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:03:54.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:03:54.538+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:03:58.158 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:03:58.157+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:04:00.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:04:00.949 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:04:00.946+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:04:00.949 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:04:00.946+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:04:01.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:01.147+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:01.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:01.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:01.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:01.157+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:01.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:04:01.528 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:04:01.543 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1023,"stamp":"2023-10-03T18:04:00.502606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22121149396,"num_objects":20522,"num_object_clones":15273,"num_object_copies":41044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20522,"num_whiteouts":0,"num_read":927368,"num_read_kb":3505009,"num_write":891832,"num_write_kb":3567303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80550,"ondisk_log_size":80550,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10642016,"kb_used_data":7130504,"kb_used_omap":368,"kb_used_meta":3511119,"kb_avail":272473504,"statfs":{"total":289910292480,"available":279012868096,"internally_reserved":0,"allocated":7301636096,"data_stored":26982618137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":377612,"internal_metadata":3595386100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,1,4,9,9,15,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":18804736,"num_objects":99,"num_object_clones":99,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":4263,"num_read_kb":16904,"num_write":4510,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.013239"},"pg_stats":[{"pgid":"2.7","version":"93'116373","reported_seq":228064,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:58.797357+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:03:58.797357+0000","last_peered":"2023-10-03T18:03:58.797357+0000","last_clean":"2023-10-03T18:03:58.797357+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:03:58.797357+0000","last_undegraded":"2023-10-03T18:03:58.797357+0000","last_fullsized":"2023-10-03T18:03:58.797357+0000","mapping_epoch":12,"log_start":"88'106316","ondisk_log_start":"88'106316","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":620,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 141s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2757362048,"num_objects":2555,"num_object_clones":1906,"num_object_copies":5110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2555,"num_whiteouts":0,"num_read":114605,"num_read_kb":437282,"num_write":111154,"num_write_kb":444511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"93'105686","reported_seq":209322,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:58.778838+0000","last_change":"2023-10-03T17:34:48.145705+0000","last_active":"2023-10-03T18:03:58.778838+0000","last_peered":"2023-10-03T18:03:58.778838+0000","last_clean":"2023-10-03T18:03:58.778838+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:03:58.778838+0000","last_undegraded":"2023-10-03T18:03:58.778838+0000","last_fullsized":"2023-10-03T18:03:58.778838+0000","mapping_epoch":12,"log_start":"88'95620","ondisk_log_start":"88'95620","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049357999999999997,"stat_sum":{"num_bytes":2546328123,"num_objects":2362,"num_object_clones":1759,"num_object_copies":4724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2362,"num_whiteouts":0,"num_read":107318,"num_read_kb":404575,"num_write":102463,"num_write_kb":409639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"93'128130","reported_seq":249511,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:58.813836+0000","last_change":"2023-10-03T17:34:48.146616+0000","last_active":"2023-10-03T18:03:58.813836+0000","last_peered":"2023-10-03T18:03:58.813836+0000","last_clean":"2023-10-03T18:03:58.813836+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:03:58.813836+0000","last_undegraded":"2023-10-03T18:03:58.813836+0000","last_fullsized":"2023-10-03T18:03:58.813836+0000","mapping_epoch":12,"log_start":"88'118120","ondisk_log_start":"88'118120","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012867919999999999,"stat_sum":{"num_bytes":2985431492,"num_objects":2764,"num_object_clones":2058,"num_object_copies":5528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":125742,"num_read_kb":472346,"num_write":120272,"num_write_kb":480899,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"93'120068","reported_seq":236499,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:59.200879+0000","last_change":"2023-10-03T18:03:58.574805+0000","last_active":"2023-10-03T18:03:59.200879+0000","last_peered":"2023-10-03T18:03:59.200879+0000","last_clean":"2023-10-03T18:03:59.200879+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:03:59.200879+0000","last_undegraded":"2023-10-03T18:03:59.200879+0000","last_fullsized":"2023-10-03T18:03:59.200879+0000","mapping_epoch":12,"log_start":"88'110016","ondisk_log_start":"88'110016","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414473,"stat_sum":{"num_bytes":2885644943,"num_objects":2676,"num_object_clones":1991,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2676,"num_whiteouts":0,"num_read":119462,"num_read_kb":455858,"num_write":116738,"num_write_kb":466829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":246,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:02.456735+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:03:02.456735+0000","last_peered":"2023-10-03T18:03:02.456735+0000","last_clean":"2023-10-03T18:03:02.456735+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:03:02.456735+0000","last_undegraded":"2023-10-03T18:03:02.456735+0000","last_fullsized":"2023-10-03T18:03:02.456735+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"93'122917","reported_seq":241632,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:58.745543+0000","last_change":"2023-10-03T17:34:48.146312+0000","last_active":"2023-10-03T18:03:58.745543+0000","last_peered":"2023-10-03T18:03:58.745543+0000","last_clean":"2023-10-03T18:03:58.745543+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:03:58.745543+0000","last_undegraded":"2023-10-03T18:03:58.745543+0000","last_fullsized":"2023-10-03T18:03:58.745543+0000","mapping_epoch":12,"log_start":"88'112820","ondisk_log_start":"88'112820","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001056888,"stat_sum":{"num_bytes":2933211826,"num_objects":2725,"num_object_clones":2029,"num_object_copies":5450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2725,"num_whiteouts":0,"num_read":122157,"num_read_kb":465274,"num_write":117957,"num_write_kb":471696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"93'111815","reported_seq":219930,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:59.218026+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:03:59.218026+0000","last_peered":"2023-10-03T18:03:59.218026+0000","last_clean":"2023-10-03T18:03:59.218026+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:03:59.218026+0000","last_undegraded":"2023-10-03T18:03:59.218026+0000","last_fullsized":"2023-10-03T18:03:59.218026+0000","mapping_epoch":12,"log_start":"88'101718","ondisk_log_start":"88'101718","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":668,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 140s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2673082839,"num_objects":2480,"num_object_clones":1848,"num_object_copies":4960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2480,"num_whiteouts":0,"num_read":111568,"num_read_kb":421324,"num_write":107549,"num_write_kb":430049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"93'111988","reported_seq":222552,"reported_epoch":93,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:03:56.117577+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:03:56.117577+0000","last_peered":"2023-10-03T18:03:56.117577+0000","last_clean":"2023-10-03T18:03:56.117577+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:03:56.117577+0000","last_undegraded":"2023-10-03T18:03:56.117577+0000","last_fullsized":"2023-10-03T18:03:56.117577+0000","mapping_epoch":12,"log_start":"88'101921","ondisk_log_start":"88'101921","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":798,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 143s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2675253920,"num_objects":2483,"num_object_clones":1843,"num_object_copies":4966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":114944,"num_read_kb":426764,"num_write":108313,"num_write_kb":432976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"93'112026","reported_seq":220522,"reported_epoch":93,"state":"active+clean","last_fresh":"2023-10-03T18:03:58.790436+0000","last_change":"2023-10-03T17:34:48.146337+0000","last_active":"2023-10-03T18:03:58.790436+0000","last_peered":"2023-10-03T18:03:58.790436+0000","last_clean":"2023-10-03T18:03:58.790436+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:03:58.790436+0000","last_undegraded":"2023-10-03T18:03:58.790436+0000","last_fullsized":"2023-10-03T18:03:58.790436+0000","mapping_epoch":12,"log_start":"88'102021","ondisk_log_start":"88'102021","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001179353,"stat_sum":{"num_bytes":2664243837,"num_objects":2475,"num_object_clones":1839,"num_object_copies":4950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2475,"num_whiteouts":0,"num_read":111466,"num_read_kb":421495,"num_write":107249,"num_write_kb":428810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22120559028,"num_objects":20520,"num_object_clones":15273,"num_object_copies":41040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20520,"num_whiteouts":0,"num_read":927262,"num_read_kb":3504918,"num_write":891695,"num_write_kb":3565409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7299956736,"data_stored":26986918760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31183,"internal_metadata":0},"log_size":80451,"ondisk_log_size":80451,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738744,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2843908,"kb_used_data":1368956,"kb_used_omap":1,"kb_used_meta":1474942,"kb_avail":91527932,"statfs":{"total":96636764160,"available":93724602368,"internally_reserved":0,"allocated":1401810944,"data_stored":5181902995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1510341062},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,0,0,3,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":2.3039999999999998,"15min":2.6560000000000001},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9870000000000001,"5min":11.130000000000001,"15min":22.562000000000001},"last":10.145},{"interface":"front","average":{"1min":1.226,"5min":2.9870000000000001,"15min":2.8300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":7.3200000000000003,"5min":14.52,"15min":21.827999999999999},"last":9.7449999999999992}]},{"osd":1,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":2.173,"15min":2.3940000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":4.0780000000000003,"5min":11.007,"15min":26.074000000000002},"last":2.4119999999999999},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.9750000000000001,"15min":2.0449999999999999},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":3.7530000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.68500000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738743,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2706540,"kb_used_data":2194916,"kb_used_omap":347,"kb_used_meta":511268,"kb_avail":91665300,"statfs":{"total":96636764160,"available":93865267200,"internally_reserved":0,"allocated":2247593984,"data_stored":8303134263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":523538608},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,5,3,5,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.657,"5min":3.8730000000000002,"15min":3.7120000000000002},"min":{"1min":0.26400000000000001,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":13.420999999999999,"5min":15.602,"15min":21.928999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":2.5649999999999999,"5min":2.4620000000000002,"15min":3.5659999999999998},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":10.092000000000001,"5min":11.981999999999999,"15min":24.215},"last":0.28499999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:02:56 2023","interfaces":[{"interface":"back","average":{"1min":3.944,"5min":2.1779999999999999,"15min":2.452},"min":{"1min":0.22500000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":14.195,"5min":14.195,"15min":14.195},"last":1.575},{"interface":"front","average":{"1min":1.889,"5min":2.129,"15min":2.3490000000000002},"min":{"1min":0.245,"5min":0.20899999999999999,"15min":0.20599999999999999},"max":{"1min":5.681,"5min":12.630000000000001,"15min":13.677},"last":0.245}]}]},{"osd":1,"up_from":8,"seq":34359738738,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5091568,"kb_used_data":3566632,"kb_used_omap":19,"kb_used_meta":1524908,"kb_avail":89280272,"statfs":{"total":96636764160,"available":91422998528,"internally_reserved":0,"allocated":3652231168,"data_stored":13497580879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19842,"internal_metadata":1561506430},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,3,4,3,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":3.1600000000000001,"15min":3.5390000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.699,"5min":12.513999999999999,"15min":39.990000000000002},"last":1.468},{"interface":"front","average":{"1min":4.0369999999999999,"5min":4.077,"15min":3.9340000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":10.377000000000001,"5min":20.716999999999999,"15min":37.381},"last":20.033000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.6219999999999999,"15min":2.585},"min":{"1min":0.24099999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.492999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":4.492},{"interface":"front","average":{"1min":2.7930000000000001,"5min":2.141,"15min":3.2629999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.7629999999999999,"5min":12.255000000000001,"15min":22.739000000000001},"last":20.108000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2247045120,"data_stored":8306432668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3651362816,"data_stored":13497790900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18248,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1401548800,"data_stored":5182695192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:04:01.544 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.544 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.544 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.545 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.545 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.545 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.545 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.545 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:01.546 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-03T18:04:01.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:01.835+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:04:01.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:01.929+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:02.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.012+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:02.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.012+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:02.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.073+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:02.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.074+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:02.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.138+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:02.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.138+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:02.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.148+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:02.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.189+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:02.208 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.208+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:02.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.219+0000 7fc515f98700 -1 reset not still connected to 0x562207692680 2023-10-03T18:04:02.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.265+0000 7fc515f98700 -1 reset not still connected to 0x562213ce1930 2023-10-03T18:04:02.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.343+0000 7fc515f98700 -1 reset not still connected to 0x562231dc92b0 2023-10-03T18:04:02.356 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-03T18:04:02.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:04:02.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.653+0000 7fc515f98700 -1 reset not still connected to 0x562213ce1930 2023-10-03T18:04:02.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:02.665+0000 7fc515f98700 -1 reset not still connected to 0x562231dc92b0 2023-10-03T18:04:03.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:03.074+0000 7fc515f98700 -1 reset not still connected to 0x56220d5bcd00 2023-10-03T18:04:03.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:03.074+0000 7fc515f98700 -1 reset not still connected to 0x562215e0d790 2023-10-03T18:04:03.098 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:04:03.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:04:03.254 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:03.251+0000 7fc515f98700 -1 reset not still connected to 0x5622183fb450 2023-10-03T18:04:03.765 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:04:03.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:04:04.460 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:04:04.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:04:05.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.096+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:05.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.112+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:05.132 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:04:05.148 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-03T18:04:05.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.173+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:05.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.184+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:05.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.267+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:05.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.467+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:05.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.467+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:05.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.468+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:05.500 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.500+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:05.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:05.507+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:05.805 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-03T18:04:05.822 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-03T18:04:06.486 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-03T18:04:06.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:04:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:04:07.168 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:04:08.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.045+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:08.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.095+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:08.144 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.143+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:08.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.144+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:08.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.218+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:08.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.218+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:08.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.249+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:08.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.268+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:08.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.329+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:08.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.712+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:08.751 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.749+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:08.751 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.749+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:08.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.768+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:08.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.810+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:08.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:08.830+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:14.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.225+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:14.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.251+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:14.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.314+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:14.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.348+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:14.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.357+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:14.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.381+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:14.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.514+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:14.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.515+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:04:14.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.516+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:14.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.516+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:14.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.535+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:14.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.610+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:14.623 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.622+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:14.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.665+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:14.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.665+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:14.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.697+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:14.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.699+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:14.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.844+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:14.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.845+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:04:14.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.845+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:14.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.845+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:14.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.853+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:14.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.919+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:14.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.938+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:14.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.968+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:14.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.968+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:14.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.968+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:14.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:14.989+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:15.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:15.054+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:18.828 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.827+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:18.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.827+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:18.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.873+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:18.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.873+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:18.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.901+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:18.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.917+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:18.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.917+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:18.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.934+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:18.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.934+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:18.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:18.934+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:23.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.127+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:23.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.255+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:23.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.256+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:23.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.256+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:23.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.306+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:23.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.327+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:23.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:23.509+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:25.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:25.067+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:25.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:25.067+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:27.169 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:04:28.040 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:04:28.046 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:04:28.067 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1039,"stamp":"2023-10-03T18:04:26.519654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22154819377,"num_objects":23762,"num_object_clones":18496,"num_object_copies":47524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23762,"num_whiteouts":0,"num_read":937452,"num_read_kb":3536814,"num_write":899957,"num_write_kb":3599491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80435,"ondisk_log_size":80435,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11007992,"kb_used_data":7197984,"kb_used_omap":368,"kb_used_meta":3809615,"kb_avail":272107528,"statfs":{"total":289910292480,"available":278638108672,"internally_reserved":0,"allocated":7370735616,"data_stored":31668023719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":377612,"internal_metadata":3901046004},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,12,3,5,9,17,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19668992,"num_objects":1870,"num_object_clones":1870,"num_object_copies":3740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1870,"num_whiteouts":0,"num_read":4454,"num_read_kb":17624,"num_write":4680,"num_write_kb":18720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006289"},"pg_stats":[{"pgid":"2.7","version":"97'117989","reported_seq":230234,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:26.276735+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:04:26.276735+0000","last_peered":"2023-10-03T18:04:26.276735+0000","last_clean":"2023-10-03T18:04:26.276735+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:04:26.276735+0000","last_undegraded":"2023-10-03T18:04:26.276735+0000","last_fullsized":"2023-10-03T18:04:26.276735+0000","mapping_epoch":12,"log_start":"88'107923","ondisk_log_start":"88'107923","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":831,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 168s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2761466391,"num_objects":2999,"num_object_clones":2347,"num_object_copies":5998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":115743,"num_read_kb":441032,"num_write":112134,"num_write_kb":448404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'107208","reported_seq":211396,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:26.269204+0000","last_change":"2023-10-03T18:04:14.118014+0000","last_active":"2023-10-03T18:04:26.269204+0000","last_peered":"2023-10-03T18:04:26.269204+0000","last_clean":"2023-10-03T18:04:26.269204+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:04:26.269204+0000","last_undegraded":"2023-10-03T18:04:26.269204+0000","last_fullsized":"2023-10-03T18:04:26.269204+0000","mapping_epoch":12,"log_start":"88'97125","ondisk_log_start":"88'97125","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":80,"snaptrim_duration":10.501562221,"stat_sum":{"num_bytes":2550092348,"num_objects":2708,"num_object_clones":2105,"num_object_copies":5416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2708,"num_whiteouts":0,"num_read":108417,"num_read_kb":408219,"num_write":103382,"num_write_kb":413279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.5","version":"97'130035","reported_seq":252080,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:26.270298+0000","last_change":"2023-10-03T18:04:16.919412+0000","last_active":"2023-10-03T18:04:26.270298+0000","last_peered":"2023-10-03T18:04:26.270298+0000","last_clean":"2023-10-03T18:04:26.270298+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:04:26.270298+0000","last_undegraded":"2023-10-03T18:04:26.270298+0000","last_fullsized":"2023-10-03T18:04:26.270298+0000","mapping_epoch":12,"log_start":"88'120029","ondisk_log_start":"88'120029","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":84,"snaptrim_duration":13.342356133999999,"stat_sum":{"num_bytes":2990150235,"num_objects":3169,"num_object_clones":2460,"num_object_copies":6338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":127262,"num_read_kb":476866,"num_write":121395,"num_write_kb":485346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"97'121660","reported_seq":238555,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:24.299403+0000","last_change":"2023-10-03T18:04:23.923007+0000","last_active":"2023-10-03T18:04:24.299403+0000","last_peered":"2023-10-03T18:04:24.299403+0000","last_clean":"2023-10-03T18:04:24.299403+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:04:24.299403+0000","last_undegraded":"2023-10-03T18:04:24.299403+0000","last_fullsized":"2023-10-03T18:04:24.299403+0000","mapping_epoch":12,"log_start":"88'111621","ondisk_log_start":"88'111621","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":61,"snaptrim_duration":5.1287665809999998,"stat_sum":{"num_bytes":2889442037,"num_objects":3046,"num_object_clones":2359,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":120573,"num_read_kb":459521,"num_write":117659,"num_write_kb":470482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":250,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T18:04:02.624811+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:04:02.624811+0000","last_peered":"2023-10-03T18:04:02.624811+0000","last_clean":"2023-10-03T18:04:02.624811+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:04:02.624811+0000","last_undegraded":"2023-10-03T18:04:02.624811+0000","last_fullsized":"2023-10-03T18:04:02.624811+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'124835","reported_seq":244068,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:26.263302+0000","last_change":"2023-10-03T18:04:18.644296+0000","last_active":"2023-10-03T18:04:26.263302+0000","last_peered":"2023-10-03T18:04:26.263302+0000","last_clean":"2023-10-03T18:04:26.263302+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:04:26.263302+0000","last_undegraded":"2023-10-03T18:04:26.263302+0000","last_fullsized":"2023-10-03T18:04:26.263302+0000","mapping_epoch":12,"log_start":"88'114828","ondisk_log_start":"88'114828","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":103,"snaptrim_duration":15.067612171,"stat_sum":{"num_bytes":2937983818,"num_objects":3148,"num_object_clones":2449,"num_object_copies":6296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3148,"num_whiteouts":0,"num_read":123418,"num_read_kb":469458,"num_write":119111,"num_write_kb":476265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"97'113330","reported_seq":221956,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:24.283903+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:04:24.283903+0000","last_peered":"2023-10-03T18:04:24.283903+0000","last_clean":"2023-10-03T18:04:24.283903+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:04:24.283903+0000","last_undegraded":"2023-10-03T18:04:24.283903+0000","last_fullsized":"2023-10-03T18:04:24.283903+0000","mapping_epoch":12,"log_start":"88'103322","ondisk_log_start":"88'103322","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":880,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 165s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2676798062,"num_objects":2925,"num_object_clones":2290,"num_object_copies":5850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2925,"num_whiteouts":0,"num_read":112748,"num_read_kb":424895,"num_write":108449,"num_write_kb":433613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"97'113800","reported_seq":225172,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:26.187297+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:04:26.187297+0000","last_peered":"2023-10-03T18:04:26.187297+0000","last_clean":"2023-10-03T18:04:26.187297+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:04:26.187297+0000","last_undegraded":"2023-10-03T18:04:26.187297+0000","last_fullsized":"2023-10-03T18:04:26.187297+0000","mapping_epoch":12,"log_start":"88'103727","ondisk_log_start":"88'103727","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1044,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 173s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2680042296,"num_objects":2943,"num_object_clones":2300,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":116528,"num_read_kb":431362,"num_write":109469,"num_write_kb":437535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"97'113680","reported_seq":222729,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:26.222539+0000","last_change":"2023-10-03T18:04:12.747544+0000","last_active":"2023-10-03T18:04:26.222539+0000","last_peered":"2023-10-03T18:04:26.222539+0000","last_clean":"2023-10-03T18:04:26.222539+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:04:26.222539+0000","last_undegraded":"2023-10-03T18:04:26.222539+0000","last_fullsized":"2023-10-03T18:04:26.222539+0000","mapping_epoch":12,"log_start":"88'103626","ondisk_log_start":"88'103626","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":71,"snaptrim_duration":9.1152471940000002,"stat_sum":{"num_bytes":2668253822,"num_objects":2822,"num_object_clones":2186,"num_object_copies":5644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2822,"num_whiteouts":0,"num_read":112657,"num_read_kb":425370,"num_write":108221,"num_write_kb":432673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22154229009,"num_objects":23760,"num_object_clones":18496,"num_object_copies":47520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23760,"num_whiteouts":0,"num_read":937346,"num_read_kb":3536723,"num_write":899820,"num_write_kb":3597597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7368056832,"data_stored":31708998178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31183,"internal_metadata":0},"log_size":80336,"ondisk_log_size":80336,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738749,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2901268,"kb_used_data":1380492,"kb_used_omap":1,"kb_used_meta":1520766,"kb_avail":91470572,"statfs":{"total":96636764160,"available":93665865728,"internally_reserved":0,"allocated":1413623808,"data_stored":6048156100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1557264838},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,0,0,1,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":2.3039999999999998,"15min":2.6560000000000001},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9870000000000001,"5min":11.130000000000001,"15min":22.562000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.226,"5min":2.9870000000000001,"15min":2.8300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":7.3200000000000003,"5min":14.52,"15min":21.827999999999999},"last":0.94899999999999995}]},{"osd":1,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":2.173,"15min":2.3940000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":4.0780000000000003,"5min":11.007,"15min":26.074000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.9750000000000001,"15min":2.0449999999999999},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":3.7530000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.25600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738749,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2819904,"kb_used_data":2218040,"kb_used_omap":347,"kb_used_meta":601508,"kb_avail":91551936,"statfs":{"total":96636764160,"available":93749182464,"internally_reserved":0,"allocated":2271272960,"data_stored":9748037531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":615944368},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,0,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:04:00 2023","interfaces":[{"interface":"back","average":{"1min":4.5199999999999996,"5min":3.903,"15min":3.7970000000000002},"min":{"1min":0.193,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":30.852,"5min":30.852,"15min":30.852},"last":4.5529999999999999},{"interface":"front","average":{"1min":2.8130000000000002,"5min":2.54,"15min":3.4780000000000002},"min":{"1min":0.26700000000000002,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":16.584,"5min":16.584,"15min":24.215},"last":4.4950000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:04:00 2023","interfaces":[{"interface":"back","average":{"1min":2.3460000000000001,"5min":2.3330000000000002,"15min":2.5049999999999999},"min":{"1min":0.22800000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":11.818,"5min":14.195,"15min":14.195},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.95,"5min":1.9810000000000001,"15min":2.2970000000000002},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":6.5789999999999997,"5min":11.94,"15min":13.677},"last":0.32900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738743,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5286820,"kb_used_data":3599452,"kb_used_omap":19,"kb_used_meta":1687340,"kb_avail":89085020,"statfs":{"total":96636764160,"available":91223060480,"internally_reserved":0,"allocated":3685838848,"data_stored":15871830088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19842,"internal_metadata":1727836798},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,3,4,8,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":3.1600000000000001,"15min":3.5390000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.699,"5min":12.513999999999999,"15min":39.990000000000002},"last":6.7619999999999996},{"interface":"front","average":{"1min":4.0369999999999999,"5min":4.077,"15min":3.9340000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":10.377000000000001,"5min":20.716999999999999,"15min":37.381},"last":0.379}]},{"osd":2,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.6219999999999999,"15min":2.585},"min":{"1min":0.24099999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.492999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":2.7930000000000001,"5min":2.141,"15min":3.2629999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.7629999999999999,"5min":12.255000000000001,"15min":22.739000000000001},"last":0.41999999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2270306304,"data_stored":9766304868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3684700160,"data_stored":15883924753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18248,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1413050368,"data_stored":6058768557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:04:28.068 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.069 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.069 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.069 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.069 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.069 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.070 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.070 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:28.070 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:04:35.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.270+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:35.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.270+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:35.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.270+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:35.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.334+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:35.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.334+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:35.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:35.427+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:36.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.300+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:04:36.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.315+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.315+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.315+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.317+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.317+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.317+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:36.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.317+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:36.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.321+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:36.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:36.321+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:40.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.355+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:40.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.355+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:40.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.540+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:40.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.540+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:40.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.542+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:40.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.793+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:40.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.793+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:40.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.793+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:40.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.812+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:40.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.812+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:40.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:40.812+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:04:48.070 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:04:48.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:04:48.841 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:04:48.864 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1050,"stamp":"2023-10-03T18:04:48.534147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22190393137,"num_objects":24890,"num_object_clones":19624,"num_object_copies":49780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24890,"num_whiteouts":0,"num_read":947423,"num_read_kb":3571789,"num_write":908261,"num_write_kb":3632707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80436,"ondisk_log_size":80436,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11254300,"kb_used_data":7262404,"kb_used_omap":368,"kb_used_meta":3991503,"kb_avail":271861220,"statfs":{"total":289910292480,"available":278385889280,"internally_reserved":0,"allocated":7436701696,"data_stored":33333928359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":377612,"internal_metadata":4087299316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,8,5,1,11,11,8,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":24444928,"num_objects":566,"num_object_clones":566,"num_object_copies":1132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":566,"num_whiteouts":0,"num_read":7172,"num_read_kb":24303,"num_write":5651,"num_write_kb":22604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002345"},"pg_stats":[{"pgid":"2.7","version":"97'119267","reported_seq":232472,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:47.184164+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:04:47.184164+0000","last_peered":"2023-10-03T18:04:47.184164+0000","last_clean":"2023-10-03T18:04:47.184164+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:04:47.184164+0000","last_undegraded":"2023-10-03T18:04:47.184164+0000","last_fullsized":"2023-10-03T18:04:47.184164+0000","mapping_epoch":12,"log_start":"91'109223","ondisk_log_start":"91'109223","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":907,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 189s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2766058007,"num_objects":3152,"num_object_clones":2500,"num_object_copies":6304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3152,"num_whiteouts":0,"num_read":116953,"num_read_kb":445392,"num_write":113172,"num_write_kb":452556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"97'108288","reported_seq":213403,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:47.130027+0000","last_change":"2023-10-03T18:04:14.118014+0000","last_active":"2023-10-03T18:04:47.130027+0000","last_peered":"2023-10-03T18:04:47.130027+0000","last_clean":"2023-10-03T18:04:47.130027+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:04:47.130027+0000","last_undegraded":"2023-10-03T18:04:47.130027+0000","last_fullsized":"2023-10-03T18:04:47.130027+0000","mapping_epoch":12,"log_start":"88'98228","ondisk_log_start":"88'98228","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":80,"snaptrim_duration":10.501562221,"stat_sum":{"num_bytes":2554167868,"num_objects":2836,"num_object_clones":2233,"num_object_copies":5672,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2836,"num_whiteouts":0,"num_read":109568,"num_read_kb":412304,"num_write":104322,"num_write_kb":417039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.5","version":"97'131385","reported_seq":254487,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:47.112740+0000","last_change":"2023-10-03T18:04:16.919412+0000","last_active":"2023-10-03T18:04:47.112740+0000","last_peered":"2023-10-03T18:04:47.112740+0000","last_clean":"2023-10-03T18:04:47.112740+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:04:47.112740+0000","last_undegraded":"2023-10-03T18:04:47.112740+0000","last_fullsized":"2023-10-03T18:04:47.112740+0000","mapping_epoch":12,"log_start":"91'121332","ondisk_log_start":"91'121332","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":84,"snaptrim_duration":13.342356133999999,"stat_sum":{"num_bytes":2994963035,"num_objects":3324,"num_object_clones":2615,"num_object_copies":6648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3324,"num_whiteouts":0,"num_read":128685,"num_read_kb":481674,"num_write":122542,"num_write_kb":489934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"97'122878","reported_seq":240690,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:44.327969+0000","last_change":"2023-10-03T18:04:34.932464+0000","last_active":"2023-10-03T18:04:44.327969+0000","last_peered":"2023-10-03T18:04:44.327969+0000","last_clean":"2023-10-03T18:04:44.327969+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:04:44.327969+0000","last_undegraded":"2023-10-03T18:04:44.327969+0000","last_fullsized":"2023-10-03T18:04:44.327969+0000","mapping_epoch":12,"log_start":"89'112826","ondisk_log_start":"89'112826","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":61,"snaptrim_duration":5.1287665809999998,"stat_sum":{"num_bytes":2893902581,"num_objects":3209,"num_object_clones":2522,"num_object_copies":6418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3209,"num_whiteouts":0,"num_read":121756,"num_read_kb":463801,"num_write":118704,"num_write_kb":474662,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":250,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-03T18:04:02.624811+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:04:02.624811+0000","last_peered":"2023-10-03T18:04:02.624811+0000","last_clean":"2023-10-03T18:04:02.624811+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:04:02.624811+0000","last_undegraded":"2023-10-03T18:04:02.624811+0000","last_fullsized":"2023-10-03T18:04:02.624811+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"97'126154","reported_seq":246451,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:47.089213+0000","last_change":"2023-10-03T18:04:18.644296+0000","last_active":"2023-10-03T18:04:47.089213+0000","last_peered":"2023-10-03T18:04:47.089213+0000","last_clean":"2023-10-03T18:04:47.089213+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:04:47.089213+0000","last_undegraded":"2023-10-03T18:04:47.089213+0000","last_fullsized":"2023-10-03T18:04:47.089213+0000","mapping_epoch":12,"log_start":"91'116130","ondisk_log_start":"91'116130","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":103,"snaptrim_duration":15.067612171,"stat_sum":{"num_bytes":2943120202,"num_objects":3278,"num_object_clones":2579,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":124715,"num_read_kb":474186,"num_write":120276,"num_write_kb":480925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"97'114505","reported_seq":224008,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:44.359512+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:04:44.359512+0000","last_peered":"2023-10-03T18:04:44.359512+0000","last_clean":"2023-10-03T18:04:44.359512+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:04:44.359512+0000","last_undegraded":"2023-10-03T18:04:44.359512+0000","last_fullsized":"2023-10-03T18:04:44.359512+0000","mapping_epoch":12,"log_start":"88'104424","ondisk_log_start":"88'104424","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":944,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 185s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2680791662,"num_objects":3062,"num_object_clones":2427,"num_object_copies":6124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3062,"num_whiteouts":0,"num_read":113907,"num_read_kb":428865,"num_write":109416,"num_write_kb":437481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"97'115029","reported_seq":227360,"reported_epoch":97,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:04:46.331794+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:04:46.331794+0000","last_peered":"2023-10-03T18:04:46.331794+0000","last_clean":"2023-10-03T18:04:46.331794+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:04:46.331794+0000","last_undegraded":"2023-10-03T18:04:46.331794+0000","last_fullsized":"2023-10-03T18:04:46.331794+0000","mapping_epoch":12,"log_start":"91'105028","ondisk_log_start":"91'105028","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1164,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 193s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2684244792,"num_objects":3072,"num_object_clones":2429,"num_object_copies":6144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3072,"num_whiteouts":0,"num_read":117816,"num_read_kb":435597,"num_write":110460,"num_write_kb":441499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"97'114848","reported_seq":224919,"reported_epoch":97,"state":"active+clean","last_fresh":"2023-10-03T18:04:47.121420+0000","last_change":"2023-10-03T18:04:12.747544+0000","last_active":"2023-10-03T18:04:47.121420+0000","last_peered":"2023-10-03T18:04:47.121420+0000","last_clean":"2023-10-03T18:04:47.121420+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:04:47.121420+0000","last_undegraded":"2023-10-03T18:04:47.121420+0000","last_fullsized":"2023-10-03T18:04:47.121420+0000","mapping_epoch":12,"log_start":"88'104826","ondisk_log_start":"88'104826","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":71,"snaptrim_duration":9.1152471940000002,"stat_sum":{"num_bytes":2672554622,"num_objects":2955,"num_object_clones":2319,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":113917,"num_read_kb":429879,"num_write":109232,"num_write_kb":436717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22189802769,"num_objects":24888,"num_object_clones":19624,"num_object_copies":49776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24888,"num_whiteouts":0,"num_read":947317,"num_read_kb":3571698,"num_write":908124,"num_write_kb":3630813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7435788288,"data_stored":33359833634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31183,"internal_metadata":0},"log_size":80337,"ondisk_log_size":80337,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738753,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2944084,"kb_used_data":1392780,"kb_used_omap":1,"kb_used_meta":1551294,"kb_avail":91427756,"statfs":{"total":96636764160,"available":93622022144,"internally_reserved":0,"allocated":1426206720,"data_stored":6385748420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1588525510},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":2.7349999999999999,"5min":2.3039999999999998,"15min":2.6560000000000001},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9870000000000001,"5min":11.130000000000001,"15min":22.562000000000001},"last":9.2710000000000008},{"interface":"front","average":{"1min":1.226,"5min":2.9870000000000001,"15min":2.8300000000000001},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":7.3200000000000003,"5min":14.52,"15min":21.827999999999999},"last":3.5459999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:03:45 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":2.173,"15min":2.3940000000000001},"min":{"1min":0.22,"5min":0.22,"15min":0.19600000000000001},"max":{"1min":4.0780000000000003,"5min":11.007,"15min":26.074000000000002},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.94799999999999995,"5min":1.9750000000000001,"15min":2.0449999999999999},"min":{"1min":0.22,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":3.7530000000000001,"5min":16.457000000000001,"15min":16.457000000000001},"last":3.379}]}]},{"osd":0,"up_from":8,"seq":34359738753,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2889696,"kb_used_data":2237848,"kb_used_omap":347,"kb_used_meta":651492,"kb_avail":91482144,"statfs":{"total":96636764160,"available":93677715456,"internally_reserved":0,"allocated":2291556352,"data_stored":10272677787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":667127984},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:04:00 2023","interfaces":[{"interface":"back","average":{"1min":4.5199999999999996,"5min":3.903,"15min":3.7970000000000002},"min":{"1min":0.193,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":30.852,"5min":30.852,"15min":30.852},"last":7.851},{"interface":"front","average":{"1min":2.8130000000000002,"5min":2.54,"15min":3.4780000000000002},"min":{"1min":0.26700000000000002,"5min":0.20499999999999999,"15min":0.19400000000000001},"max":{"1min":16.584,"5min":16.584,"15min":24.215},"last":8.0950000000000006}]},{"osd":2,"last update":"Tue Oct 3 18:04:00 2023","interfaces":[{"interface":"back","average":{"1min":2.3460000000000001,"5min":2.3330000000000002,"15min":2.5049999999999999},"min":{"1min":0.22800000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":11.818,"5min":14.195,"15min":14.195},"last":0.29899999999999999},{"interface":"front","average":{"1min":1.95,"5min":1.9810000000000001,"15min":2.2970000000000002},"min":{"1min":0.23799999999999999,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":6.5789999999999997,"5min":11.94,"15min":13.677},"last":0.34799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738747,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5420520,"kb_used_data":3631776,"kb_used_omap":19,"kb_used_meta":1788716,"kb_avail":88951320,"statfs":{"total":96636764160,"available":91086151680,"internally_reserved":0,"allocated":3718938624,"data_stored":16675502152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19842,"internal_metadata":1831645822},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,8,2,1,5,6,5,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":1.9490000000000001,"5min":3.1600000000000001,"15min":3.5390000000000001},"min":{"1min":0.16300000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":10.699,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.23000000000000001},{"interface":"front","average":{"1min":4.0369999999999999,"5min":4.077,"15min":3.9340000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":10.377000000000001,"5min":20.716999999999999,"15min":37.381},"last":2.8029999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:03:44 2023","interfaces":[{"interface":"back","average":{"1min":3.1720000000000002,"5min":3.6219999999999999,"15min":2.585},"min":{"1min":0.24099999999999999,"5min":0.21199999999999999,"15min":0.17299999999999999},"max":{"1min":17.492999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":0.21099999999999999},{"interface":"front","average":{"1min":2.7930000000000001,"5min":2.141,"15min":3.2629999999999999},"min":{"1min":0.23300000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":7.7629999999999999,"5min":12.255000000000001,"15min":22.739000000000001},"last":2.8450000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2290667520,"data_stored":10278378596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3719258112,"data_stored":16692032785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18248,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1425862656,"data_stored":6389422253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:04:48.866 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.866 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.867 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.867 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.867 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.867 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.867 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.868 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:04:48.868 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:04:51.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.481+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:04:51.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.481+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:04:51.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.481+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:04:51.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.560+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:04:51.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.573+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:04:51.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:04:51.573+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:00.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.146+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:00.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.161+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:00.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.161+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:00.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.191+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:00.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.194+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:00.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.220+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:00.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.393+0000 7fc515f98700 -1 reset not still connected to 0x56221a26dee0 2023-10-03T18:05:00.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.394+0000 7fc515f98700 -1 reset not still connected to 0x56222dd69ee0 2023-10-03T18:05:00.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:00.394+0000 7fc515f98700 -1 reset not still connected to 0x562233b77ba0 2023-10-03T18:05:01.373 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:01.367+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:01.373 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:01.367+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:01.373 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:01.367+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:02.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.134+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:02.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.134+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:02.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:02.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.162+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:02.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.172+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:02.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.172+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:02.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.172+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:02.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.172+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:02.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.190+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:02.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:02.190+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:03.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.639+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:03.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.682+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:03.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.767+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:03.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.767+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:03.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.782+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:03.823 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.820+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:03.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:03.913+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:04.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.349+0000 7fc515f98700 -1 reset not still connected to 0x562207022b60 2023-10-03T18:05:04.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.383+0000 7fc515f98700 -1 reset not still connected to 0x562209a78c30 2023-10-03T18:05:04.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.443+0000 7fc515f98700 -1 reset not still connected to 0x5622168bb930 2023-10-03T18:05:04.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.541+0000 7fc515f98700 -1 reset not still connected to 0x56223658a410 2023-10-03T18:05:04.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.940+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:04.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.940+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:04.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:04.987+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:05.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.125+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:05.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.226+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:05.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.226+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:05.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.227+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:05.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.227+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:05.377 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.377+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:05.464 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:05.464+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:05:05.477 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:05.477+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:05.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.739+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:05.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.900+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:05.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:05.900+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:06.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.028+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:06.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.063+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:06.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.063+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:06.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.103+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:06.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.103+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:06.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:06.150+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:08.098 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.097+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:08.253 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.253+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:08.253 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.253+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:08.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.272+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:08.409 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.407+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:08.409 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.407+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:08.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.440+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:08.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.481+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:08.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:08.481+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:08.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:05:09.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:05:09.560 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:05:09.581 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1068,"stamp":"2023-10-03T18:05:08.551617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22220789604,"num_objects":25347,"num_object_clones":20080,"num_object_copies":50694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25347,"num_whiteouts":0,"num_read":957029,"num_read_kb":3602148,"num_write":915438,"num_write_kb":3660918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80465,"ondisk_log_size":80465,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11438820,"kb_used_data":7321200,"kb_used_omap":368,"kb_used_meta":4117199,"kb_avail":271676700,"statfs":{"total":289910292480,"available":278196940800,"internally_reserved":0,"allocated":7496908800,"data_stored":34113560326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":377612,"internal_metadata":4216012020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,5,3,9,11,10,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":786382,"num_objects":100,"num_object_clones":101,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":500,"num_read_kb":937,"num_write":204,"num_write_kb":718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004119"},"pg_stats":[{"pgid":"2.7","version":"100'120275","reported_seq":234327,"reported_epoch":100,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:03.111453+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:05:03.111453+0000","last_peered":"2023-10-03T18:05:03.111453+0000","last_clean":"2023-10-03T18:05:03.111453+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:05:03.111453+0000","last_undegraded":"2023-10-03T18:05:03.111453+0000","last_fullsized":"2023-10-03T18:05:03.111453+0000","mapping_epoch":12,"log_start":"91'110224","ondisk_log_start":"91'110224","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":981,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 205s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2769916439,"num_objects":3213,"num_object_clones":2561,"num_object_copies":6426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":118007,"num_read_kb":448923,"num_write":114046,"num_write_kb":456003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"100'109143","reported_seq":215091,"reported_epoch":100,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:03.002318+0000","last_change":"2023-10-03T18:05:01.706894+0000","last_active":"2023-10-03T18:05:03.002318+0000","last_peered":"2023-10-03T18:05:03.002318+0000","last_clean":"2023-10-03T18:05:03.002318+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:05:03.002318+0000","last_undegraded":"2023-10-03T18:05:03.002318+0000","last_fullsized":"2023-10-03T18:05:03.002318+0000","mapping_epoch":12,"log_start":"88'99128","ondisk_log_start":"88'99128","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":10.501562221,"stat_sum":{"num_bytes":2557395567,"num_objects":2859,"num_object_clones":2255,"num_object_copies":5718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2859,"num_whiteouts":0,"num_read":110554,"num_read_kb":415582,"num_write":105078,"num_write_kb":420031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.5","version":"100'132320","reported_seq":256394,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:02.899999+0000","last_change":"2023-10-03T18:05:01.671330+0000","last_active":"2023-10-03T18:05:02.899999+0000","last_peered":"2023-10-03T18:05:02.899999+0000","last_clean":"2023-10-03T18:05:02.899999+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:05:02.899999+0000","last_undegraded":"2023-10-03T18:05:02.899999+0000","last_fullsized":"2023-10-03T18:05:02.899999+0000","mapping_epoch":12,"log_start":"91'122234","ondisk_log_start":"91'122234","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":13.342356133999999,"stat_sum":{"num_bytes":2998465065,"num_objects":3370,"num_object_clones":2662,"num_object_copies":6740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":129990,"num_read_kb":485628,"num_write":123402,"num_write_kb":493296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.0","version":"101'124128","reported_seq":243005,"reported_epoch":101,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:04.389601+0000","last_change":"2023-10-03T18:05:02.968481+0000","last_active":"2023-10-03T18:05:04.389601+0000","last_peered":"2023-10-03T18:05:04.389601+0000","last_clean":"2023-10-03T18:05:04.389601+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:05:04.389601+0000","last_undegraded":"2023-10-03T18:05:04.389601+0000","last_fullsized":"2023-10-03T18:05:04.389601+0000","mapping_epoch":12,"log_start":"91'114126","ondisk_log_start":"91'114126","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.29989444999999998,"stat_sum":{"num_bytes":2898404085,"num_objects":3250,"num_object_clones":2563,"num_object_copies":6500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":123113,"num_read_kb":468305,"num_write":119773,"num_write_kb":478884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"58","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":253,"reported_epoch":99,"state":"active+clean","last_fresh":"2023-10-03T18:05:01.693111+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:05:01.693111+0000","last_peered":"2023-10-03T18:05:01.693111+0000","last_clean":"2023-10-03T18:05:01.693111+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:05:01.693111+0000","last_undegraded":"2023-10-03T18:05:01.693111+0000","last_fullsized":"2023-10-03T18:05:01.693111+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"100'127106","reported_seq":248309,"reported_epoch":100,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:02.901535+0000","last_change":"2023-10-03T18:05:01.664020+0000","last_active":"2023-10-03T18:05:02.901535+0000","last_peered":"2023-10-03T18:05:02.901535+0000","last_clean":"2023-10-03T18:05:02.901535+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:05:02.901535+0000","last_undegraded":"2023-10-03T18:05:02.901535+0000","last_fullsized":"2023-10-03T18:05:02.901535+0000","mapping_epoch":12,"log_start":"91'117030","ondisk_log_start":"91'117030","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":15.067612171,"stat_sum":{"num_bytes":2946724682,"num_objects":3295,"num_object_clones":2596,"num_object_copies":6590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3295,"num_whiteouts":0,"num_read":125812,"num_read_kb":477937,"num_write":121136,"num_write_kb":484329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]},{"pgid":"2.1","version":"101'115698","reported_seq":226202,"reported_epoch":101,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:04.371080+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:05:04.371080+0000","last_peered":"2023-10-03T18:05:04.371080+0000","last_clean":"2023-10-03T18:05:04.371080+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:05:04.371080+0000","last_undegraded":"2023-10-03T18:05:04.371080+0000","last_fullsized":"2023-10-03T18:05:04.371080+0000","mapping_epoch":12,"log_start":"91'105625","ondisk_log_start":"91'105625","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1049,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 205s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2685084320,"num_objects":3141,"num_object_clones":2505,"num_object_copies":6282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":115275,"num_read_kb":432984,"num_write":110434,"num_write_kb":441471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'116235","reported_seq":229576,"reported_epoch":103,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:06.432268+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:05:06.432268+0000","last_peered":"2023-10-03T18:05:06.432268+0000","last_clean":"2023-10-03T18:05:06.432268+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:05:06.432268+0000","last_undegraded":"2023-10-03T18:05:06.432268+0000","last_fullsized":"2023-10-03T18:05:06.432268+0000","mapping_epoch":12,"log_start":"91'106229","ondisk_log_start":"91'106229","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1280,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 213s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2688418566,"num_objects":3199,"num_object_clones":2557,"num_object_copies":6398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3199,"num_whiteouts":0,"num_read":119347,"num_read_kb":439745,"num_write":111440,"num_write_kb":445285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"100'115685","reported_seq":226505,"reported_epoch":100,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:02.941797+0000","last_change":"2023-10-03T18:05:01.671898+0000","last_active":"2023-10-03T18:05:02.941797+0000","last_peered":"2023-10-03T18:05:02.941797+0000","last_clean":"2023-10-03T18:05:02.941797+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:05:02.941797+0000","last_undegraded":"2023-10-03T18:05:02.941797+0000","last_fullsized":"2023-10-03T18:05:02.941797+0000","mapping_epoch":12,"log_start":"91'105628","ondisk_log_start":"91'105628","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":9.1152471940000002,"stat_sum":{"num_bytes":2675790512,"num_objects":3018,"num_object_clones":2381,"num_object_copies":6036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3018,"num_whiteouts":0,"num_read":114825,"num_read_kb":432953,"num_write":109992,"num_write_kb":439725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22220199236,"num_objects":25345,"num_object_clones":20080,"num_object_copies":50690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25345,"num_whiteouts":0,"num_read":956923,"num_read_kb":3602057,"num_write":915301,"num_write_kb":3659024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7494373376,"data_stored":34190510932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":31183,"internal_metadata":0},"log_size":80366,"ondisk_log_size":80366,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738757,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3002532,"kb_used_data":1405460,"kb_used_omap":1,"kb_used_meta":1597054,"kb_avail":91369308,"statfs":{"total":96636764160,"available":93562171392,"internally_reserved":0,"allocated":1439191040,"data_stored":6494958524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1635383750},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,4,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:04:46 2023","interfaces":[{"interface":"back","average":{"1min":3.1989999999999998,"5min":2.282,"15min":2.6960000000000002},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":10.145,"5min":10.962999999999999,"15min":22.562000000000001},"last":5.9450000000000003},{"interface":"front","average":{"1min":3.0369999999999999,"5min":3.0019999999999998,"15min":2.8809999999999998},"min":{"1min":0.23899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":12.414999999999999,"5min":14.52,"15min":21.827999999999999},"last":6.2690000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":2.0870000000000002,"15min":2.3279999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.8369999999999997,"5min":11.007,"15min":26.074000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":2.0230000000000001,"15min":1.9650000000000001},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":4.0149999999999997,"5min":16.457000000000001,"15min":16.457000000000001},"last":0.437}]}]},{"osd":0,"up_from":8,"seq":34359738757,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2922736,"kb_used_data":2256992,"kb_used_omap":347,"kb_used_meta":665380,"kb_avail":91449104,"statfs":{"total":96636764160,"available":93643882496,"internally_reserved":0,"allocated":2311159808,"data_stored":10686611062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":681349296},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,0,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":4.3120000000000003,"5min":4.1219999999999999,"15min":3.9660000000000002},"min":{"1min":0.221,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":10.057,"5min":30.852,"15min":30.852},"last":1.085},{"interface":"front","average":{"1min":4.3040000000000003,"5min":3.1200000000000001,"15min":3.621},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.19400000000000001},"max":{"1min":9.7620000000000005,"5min":16.584,"15min":24.215},"last":1.7629999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.4670000000000001,"15min":2.452},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":8.2899999999999991,"5min":14.195,"15min":14.195},"last":0.83599999999999997},{"interface":"front","average":{"1min":2.855,"5min":2.3090000000000002,"15min":2.3919999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":10.949,"5min":11.94,"15min":13.677},"last":0.314}]}]},{"osd":1,"up_from":8,"seq":34359738750,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5513552,"kb_used_data":3658748,"kb_used_omap":19,"kb_used_meta":1854764,"kb_avail":88858288,"statfs":{"total":96636764160,"available":90990886912,"internally_reserved":0,"allocated":3746557952,"data_stored":16931990740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19842,"internal_metadata":1899278974},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,1,7,7,9,11,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.9649999999999999,"15min":3.5110000000000001},"min":{"1min":0.188,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.093999999999999,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":3.6850000000000001,"5min":4.1909999999999998,"15min":3.952},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":20.033000000000001,"5min":20.716999999999999,"15min":37.381},"last":0.55400000000000005}]},{"osd":2,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":3.3860000000000001,"15min":2.4529999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":13.840999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":5.9589999999999996},{"interface":"front","average":{"1min":3.702,"5min":2.6030000000000002,"15min":3.2909999999999999},"min":{"1min":0.23400000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":20.108000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.95799999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2309885952,"data_stored":10710703307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3745918976,"data_stored":16965911876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":18248,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1438568448,"data_stored":6513895749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:05:09.583 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.583 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.584 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.584 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.584 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.584 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.585 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.585 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:09.585 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:05:10.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:10.947+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:10.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:10.955+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:10.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:10.955+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:11.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.004+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:11.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.005+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:11.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.048+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:11.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.048+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:11.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.437+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:11.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.622+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:11.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:11.629+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:11.639 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:11.637+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:11.749 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:11.748+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:05:11.750 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:11.748+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:12.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.159+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:12.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:12.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:12.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.329+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:12.414 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.411+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:12.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.514+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:12.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.514+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:12.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.845+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:12.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.845+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:12.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.864+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:12.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.864+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:12.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.865+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:12.892 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:12.891+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:13.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.017+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:13.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.017+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:13.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.055+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:13.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.124+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:13.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.124+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:13.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.161+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:13.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.171+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:13.200 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.199+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:13.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.248+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:13.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.344+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:13.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.344+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:13.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.450+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:13.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.487+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:13.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.519+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:13.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.519+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:13.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.519+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:13.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.519+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:13.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.606+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:13.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.637+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:13.858 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:13.856+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:14.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.028+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:14.088 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.087+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:14.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.129+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:14.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.162+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:14.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.162+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:14.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.170+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:14.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.241+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:14.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:14.278+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:24.655 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:24.654+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:05:24.656 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:24.654+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:05:24.676 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:24.672+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:24.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:24.685+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:05:24.692 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:24.685+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:27.298 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.296+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:27.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.374+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:27.409 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.408+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:27.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.410+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:27.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.426+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:27.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.426+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:27.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.443+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:27.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.479+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:27.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:27.479+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:29.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:05:29.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:29.649+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:29.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:29.840+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:29.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:29.874+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:30.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.308+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:30.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.327+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:30.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.328+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:30.340 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:05:30.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.334+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:30.358 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:05:30.377 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.368+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:30.377 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.368+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:30.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.376+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:30.381 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1078,"stamp":"2023-10-03T18:05:28.561575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22243821361,"num_objects":27615,"num_object_clones":22349,"num_object_copies":55230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27615,"num_whiteouts":0,"num_read":963244,"num_read_kb":3625044,"num_write":920988,"num_write_kb":3682895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80348,"ondisk_log_size":80348,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10525500,"kb_used_data":7364324,"kb_used_omap":2264,"kb_used_meta":3158887,"kb_avail":272590020,"statfs":{"total":289910292480,"available":279132180480,"internally_reserved":0,"allocated":7541067776,"data_stored":37451164255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":3234700839},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,6,15,5,14,16,11,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11943936,"num_objects":769,"num_object_clones":769,"num_object_copies":1538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":2994,"num_read_kb":11952,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007539"},"pg_stats":[{"pgid":"2.7","version":"105'121480","reported_seq":235924,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:25.607330+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:05:25.607330+0000","last_peered":"2023-10-03T18:05:25.607330+0000","last_clean":"2023-10-03T18:05:25.607330+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:05:25.607330+0000","last_undegraded":"2023-10-03T18:05:25.607330+0000","last_fullsized":"2023-10-03T18:05:25.607330+0000","mapping_epoch":12,"log_start":"91'111429","ondisk_log_start":"91'111429","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1175,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 228s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2772595223,"num_objects":3589,"num_object_clones":2937,"num_object_copies":7178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3589,"num_whiteouts":0,"num_read":118799,"num_read_kb":451821,"num_write":114704,"num_write_kb":458605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'110461","reported_seq":216679,"reported_epoch":105,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:25.601356+0000","last_change":"2023-10-03T18:05:24.557028+0000","last_active":"2023-10-03T18:05:25.601356+0000","last_peered":"2023-10-03T18:05:25.601356+0000","last_clean":"2023-10-03T18:05:25.601356+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:05:25.601356+0000","last_undegraded":"2023-10-03T18:05:25.601356+0000","last_fullsized":"2023-10-03T18:05:25.601356+0000","mapping_epoch":12,"log_start":"91'100436","ondisk_log_start":"91'100436","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":106,"snaptrim_duration":15.99223317,"stat_sum":{"num_bytes":2560193237,"num_objects":3125,"num_object_clones":2519,"num_object_copies":6250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3125,"num_whiteouts":0,"num_read":111235,"num_read_kb":418252,"num_write":105762,"num_write_kb":422754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"5"}]},{"pgid":"2.5","version":"105'133914","reported_seq":258381,"reported_epoch":105,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:25.588969+0000","last_change":"2023-10-03T18:05:25.589017+0000","last_active":"2023-10-03T18:05:25.588969+0000","last_peered":"2023-10-03T18:05:25.588969+0000","last_clean":"2023-10-03T18:05:25.588969+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:05:25.588969+0000","last_undegraded":"2023-10-03T18:05:25.588969+0000","last_fullsized":"2023-10-03T18:05:25.588969+0000","mapping_epoch":12,"log_start":"91'123840","ondisk_log_start":"91'123840","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":109,"snaptrim_duration":17.050384543,"stat_sum":{"num_bytes":3001975286,"num_objects":3683,"num_object_clones":2976,"num_object_copies":7366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3683,"num_whiteouts":0,"num_read":131033,"num_read_kb":489098,"num_write":124230,"num_write_kb":496555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"4"}]},{"pgid":"2.0","version":"105'125774","reported_seq":244872,"reported_epoch":105,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:24.500653+0000","last_change":"2023-10-03T18:05:22.980595+0000","last_active":"2023-10-03T18:05:24.500653+0000","last_peered":"2023-10-03T18:05:24.500653+0000","last_clean":"2023-10-03T18:05:24.500653+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:05:24.500653+0000","last_undegraded":"2023-10-03T18:05:24.500653+0000","last_fullsized":"2023-10-03T18:05:24.500653+0000","mapping_epoch":12,"log_start":"91'115735","ondisk_log_start":"91'115735","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":245,"snaptrim_duration":14.454624163,"stat_sum":{"num_bytes":2901205647,"num_objects":3322,"num_object_clones":2637,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":123802,"num_read_kb":471010,"num_write":120434,"num_write_kb":481509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"a"}]},{"pgid":"1.0","version":"88'99","reported_seq":258,"reported_epoch":104,"state":"active+clean","last_fresh":"2023-10-03T18:05:06.750346+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:05:06.750346+0000","last_peered":"2023-10-03T18:05:06.750346+0000","last_clean":"2023-10-03T18:05:06.750346+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:05:06.750346+0000","last_undegraded":"2023-10-03T18:05:06.750346+0000","last_fullsized":"2023-10-03T18:05:06.750346+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'128550","reported_seq":250100,"reported_epoch":105,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:25.558515+0000","last_change":"2023-10-03T18:05:24.678995+0000","last_active":"2023-10-03T18:05:25.558515+0000","last_peered":"2023-10-03T18:05:25.558515+0000","last_clean":"2023-10-03T18:05:25.558515+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:05:25.558515+0000","last_undegraded":"2023-10-03T18:05:25.558515+0000","last_fullsized":"2023-10-03T18:05:25.558515+0000","mapping_epoch":12,"log_start":"91'118540","ondisk_log_start":"91'118540","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":105,"snaptrim_duration":14.117116502,"stat_sum":{"num_bytes":2949796682,"num_objects":3600,"num_object_clones":2901,"num_object_copies":7200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3600,"num_whiteouts":0,"num_read":126676,"num_read_kb":481095,"num_write":121893,"num_write_kb":487314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"4"}]},{"pgid":"2.1","version":"105'116838","reported_seq":227645,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:24.499304+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:05:24.499304+0000","last_peered":"2023-10-03T18:05:24.499304+0000","last_clean":"2023-10-03T18:05:24.499304+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:05:24.499304+0000","last_undegraded":"2023-10-03T18:05:24.499304+0000","last_fullsized":"2023-10-03T18:05:24.499304+0000","mapping_epoch":12,"log_start":"91'106828","ondisk_log_start":"91'106828","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1198,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 225s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2687816301,"num_objects":3503,"num_object_clones":2868,"num_object_copies":7006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":115917,"num_read_kb":435497,"num_write":111081,"num_write_kb":444033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"105'117336","reported_seq":231039,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:26.537589+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:05:26.537589+0000","last_peered":"2023-10-03T18:05:26.537589+0000","last_clean":"2023-10-03T18:05:26.537589+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:05:26.537589+0000","last_undegraded":"2023-10-03T18:05:26.537589+0000","last_fullsized":"2023-10-03T18:05:26.537589+0000","mapping_epoch":12,"log_start":"91'107331","ondisk_log_start":"91'107331","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1494,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 233s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2690921222,"num_objects":3520,"num_object_clones":2878,"num_object_copies":7040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3520,"num_whiteouts":0,"num_read":119999,"num_read_kb":442342,"num_write":112037,"num_write_kb":447673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"105'117070","reported_seq":228218,"reported_epoch":105,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:25.544101+0000","last_change":"2023-10-03T18:05:22.890014+0000","last_active":"2023-10-03T18:05:25.544101+0000","last_peered":"2023-10-03T18:05:25.544101+0000","last_clean":"2023-10-03T18:05:25.544101+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:05:25.544101+0000","last_undegraded":"2023-10-03T18:05:25.544101+0000","last_fullsized":"2023-10-03T18:05:25.544101+0000","mapping_epoch":12,"log_start":"91'107035","ondisk_log_start":"91'107035","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":100,"snaptrim_duration":14.349657061,"stat_sum":{"num_bytes":2678727395,"num_objects":3271,"num_object_clones":2633,"num_object_copies":6542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3271,"num_whiteouts":0,"num_read":115677,"num_read_kb":435838,"num_write":110710,"num_write_kb":442558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22243230993,"num_objects":27613,"num_object_clones":22349,"num_object_copies":55226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27613,"num_whiteouts":0,"num_read":963138,"num_read_kb":3624953,"num_write":920851,"num_write_kb":3681001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7538921472,"data_stored":37510638114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38147,"internal_metadata":0},"log_size":80249,"ondisk_log_size":80249,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738761,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3056260,"kb_used_data":1413372,"kb_used_omap":1,"kb_used_meta":1642878,"kb_avail":91315580,"statfs":{"total":96636764160,"available":93507153920,"internally_reserved":0,"allocated":1447292928,"data_stored":7058734149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1682307526},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,5,8,0,4,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:04:46 2023","interfaces":[{"interface":"back","average":{"1min":3.1989999999999998,"5min":2.282,"15min":2.6960000000000002},"min":{"1min":0.20399999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":10.145,"5min":10.962999999999999,"15min":22.562000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":3.0369999999999999,"5min":3.0019999999999998,"15min":2.8809999999999998},"min":{"1min":0.23899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":12.414999999999999,"5min":14.52,"15min":21.827999999999999},"last":0.29699999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:04:46 2023","interfaces":[{"interface":"back","average":{"1min":1.5029999999999999,"5min":2.0870000000000002,"15min":2.3279999999999998},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":4.8369999999999997,"5min":11.007,"15min":26.074000000000002},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":2.0230000000000001,"15min":1.9650000000000001},"min":{"1min":0.251,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":4.0149999999999997,"5min":16.457000000000001,"15min":16.457000000000001},"last":1}]}]},{"osd":0,"up_from":8,"seq":34359738761,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3058764,"kb_used_data":2269828,"kb_used_omap":347,"kb_used_meta":788580,"kb_avail":91313076,"statfs":{"total":96636764160,"available":93504589824,"internally_reserved":0,"allocated":2324303872,"data_stored":11694953194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":807506096},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":4.3120000000000003,"5min":4.1219999999999999,"15min":3.9660000000000002},"min":{"1min":0.221,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":10.057,"5min":30.852,"15min":30.852},"last":0.66100000000000003},{"interface":"front","average":{"1min":4.3040000000000003,"5min":3.1200000000000001,"15min":3.621},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.19400000000000001},"max":{"1min":9.7620000000000005,"5min":16.584,"15min":24.215},"last":0.63600000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.4670000000000001,"15min":2.452},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":8.2899999999999991,"5min":14.195,"15min":14.195},"last":0.34499999999999997},{"interface":"front","average":{"1min":2.855,"5min":2.3090000000000002,"15min":2.3919999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":10.949,"5min":11.94,"15min":13.677},"last":2.0720000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738754,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4410476,"kb_used_data":3681124,"kb_used_omap":1915,"kb_used_meta":727428,"kb_avail":89961364,"statfs":{"total":96636764160,"available":92120436736,"internally_reserved":0,"allocated":3769470976,"data_stored":18697476912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":744887217},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,9,10,7,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.9649999999999999,"15min":3.5110000000000001},"min":{"1min":0.188,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.093999999999999,"5min":12.513999999999999,"15min":39.990000000000002},"last":2.7930000000000001},{"interface":"front","average":{"1min":3.6850000000000001,"5min":4.1909999999999998,"15min":3.952},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":20.033000000000001,"5min":20.716999999999999,"15min":37.381},"last":2.6080000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":3.3860000000000001,"15min":2.4529999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":13.840999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":2.6509999999999998},{"interface":"front","average":{"1min":3.702,"5min":2.6030000000000002,"15min":3.2909999999999999},"min":{"1min":0.23400000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":20.108000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.25600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2322980864,"data_stored":11698478283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3769384960,"data_stored":18748069137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1446555648,"data_stored":7064090694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:05:30.383 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.383 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.384 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.384 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.384 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.384 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.384 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.385 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:30.385 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:05:30.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.385+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:30.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:30.432+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:31.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:31.882+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:31.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:31.882+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:31.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:31.886+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:31.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:31.886+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:32.528 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:32.526+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:32.528 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:32.527+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:05:32.541 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:32.534+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:33.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.179+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:33.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.344+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:33.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.389+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:33.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.390+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:33.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.479+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:33.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.479+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:33.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:33.546+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:34.287 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.286+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:34.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.286+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:34.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.286+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:34.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.304+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:34.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.325+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:34.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.346+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:34.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.361+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:34.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.361+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:34.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.362+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:34.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.377+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:34.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:34.377+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:37.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:37.254+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:37.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:37.274+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:37.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:37.429+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:37.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:37.429+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:39.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.314+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:39.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.352+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:39.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.354+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:39.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.435+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:39.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.435+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:39.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.452+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:39.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.452+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:39.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:39.566+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:41.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:41.995+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:41.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:41.995+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:42.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.003+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:42.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.009+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:42.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.009+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:42.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:42.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:42.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:42.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:42.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:42.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.026+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:42.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.133+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:42.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.133+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:42.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.145+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:42.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.184+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:42.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.184+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:42.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.184+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:42.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.227+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:42.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.227+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:42.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:42.227+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:43.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.080+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:43.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.117+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:43.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.173+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:43.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.229+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:43.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.308+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:43.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.425+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:43.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.459+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:43.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.459+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:43.473 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.468+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:43.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.487+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:43.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.499+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:43.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:43.916+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:44.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:44.014+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:44.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:44.014+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:44.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:44.180+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:44.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:44.319+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:44.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:44.345+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:45.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.143+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:45.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.143+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:45.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.211+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:45.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.257+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:45.283 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.282+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:45.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.286+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:45.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.286+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:45.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:45.327+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:49.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.277+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:49.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.277+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:49.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.277+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:49.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.400+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:49.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.418+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:49.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.418+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:49.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.420+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:49.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.432+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:49.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:49.460+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:50.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:05:50.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:50.616+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:50.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:50.616+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:50.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:50.693+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:50.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:50.714+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:50.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:50.721+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:05:51.191 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:05:51.221 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1089,"stamp":"2023-10-03T18:05:50.579438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22271551281,"num_objects":27850,"num_object_clones":22584,"num_object_copies":55700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27850,"num_whiteouts":0,"num_read":971831,"num_read_kb":3654703,"num_write":927715,"num_write_kb":3709803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80550,"ondisk_log_size":80550,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10798500,"kb_used_data":7417612,"kb_used_omap":2264,"kb_used_meta":3378599,"kb_avail":272317020,"statfs":{"total":289910292480,"available":278852628480,"internally_reserved":0,"allocated":7595634688,"data_stored":37883951711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":3459685927},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,5,14,23,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15249408,"num_objects":31,"num_object_clones":31,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":5398,"num_read_kb":16931,"num_write":3716,"num_write_kb":14864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008690"},"pg_stats":[{"pgid":"2.7","version":"105'122630","reported_seq":237804,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:47.082041+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:05:47.082041+0000","last_peered":"2023-10-03T18:05:47.082041+0000","last_clean":"2023-10-03T18:05:47.082041+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:05:47.082041+0000","last_undegraded":"2023-10-03T18:05:47.082041+0000","last_fullsized":"2023-10-03T18:05:47.082041+0000","mapping_epoch":12,"log_start":"91'112532","ondisk_log_start":"91'112532","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1336,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 249s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2775994903,"num_objects":3750,"num_object_clones":3098,"num_object_copies":7500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3750,"num_whiteouts":0,"num_read":119775,"num_read_kb":455297,"num_write":115528,"num_write_kb":461901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"105'111719","reported_seq":218625,"reported_epoch":105,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:47.074660+0000","last_change":"2023-10-03T18:05:46.363646+0000","last_active":"2023-10-03T18:05:47.074660+0000","last_peered":"2023-10-03T18:05:47.074660+0000","last_clean":"2023-10-03T18:05:47.074660+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:05:47.074660+0000","last_undegraded":"2023-10-03T18:05:47.074660+0000","last_fullsized":"2023-10-03T18:05:47.074660+0000","mapping_epoch":12,"log_start":"91'101641","ondisk_log_start":"91'101641","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":250,"snaptrim_duration":35.246227711000003,"stat_sum":{"num_bytes":2563609301,"num_objects":3119,"num_object_clones":2513,"num_object_copies":6238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3119,"num_whiteouts":0,"num_read":112209,"num_read_kb":421681,"num_write":106579,"num_write_kb":426022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"a"}]},{"pgid":"2.5","version":"105'135322","reported_seq":260565,"reported_epoch":105,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:47.049146+0000","last_change":"2023-10-03T18:05:46.349836+0000","last_active":"2023-10-03T18:05:47.049146+0000","last_peered":"2023-10-03T18:05:47.049146+0000","last_clean":"2023-10-03T18:05:47.049146+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:05:47.049146+0000","last_undegraded":"2023-10-03T18:05:47.049146+0000","last_fullsized":"2023-10-03T18:05:47.049146+0000","mapping_epoch":12,"log_start":"93'125243","ondisk_log_start":"93'125243","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":290,"snaptrim_duration":37.811202538000003,"stat_sum":{"num_bytes":3005460982,"num_objects":3652,"num_object_clones":2945,"num_object_copies":7304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3652,"num_whiteouts":0,"num_read":132225,"num_read_kb":493014,"num_write":125074,"num_write_kb":499931,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"9"}]},{"pgid":"2.0","version":"105'127571","reported_seq":247516,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-03T18:05:49.763099+0000","last_change":"2023-10-03T18:05:49.284792+0000","last_active":"2023-10-03T18:05:49.763099+0000","last_peered":"2023-10-03T18:05:49.763099+0000","last_clean":"2023-10-03T18:05:49.763099+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:05:49.763099+0000","last_undegraded":"2023-10-03T18:05:49.763099+0000","last_fullsized":"2023-10-03T18:05:49.763099+0000","mapping_epoch":12,"log_start":"93'117540","ondisk_log_start":"93'117540","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":556,"snaptrim_duration":33.343104414999999,"stat_sum":{"num_bytes":2905227919,"num_objects":3193,"num_object_clones":2508,"num_object_copies":6386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3193,"num_whiteouts":0,"num_read":125044,"num_read_kb":475538,"num_write":121413,"num_write_kb":485425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":259,"reported_epoch":105,"state":"active+clean","last_fresh":"2023-10-03T18:05:08.603153+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:05:08.603153+0000","last_peered":"2023-10-03T18:05:08.603153+0000","last_clean":"2023-10-03T18:05:08.603153+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:05:08.603153+0000","last_undegraded":"2023-10-03T18:05:08.603153+0000","last_fullsized":"2023-10-03T18:05:08.603153+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"105'129950","reported_seq":252194,"reported_epoch":105,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:05:46.982469+0000","last_change":"2023-10-03T18:05:46.585177+0000","last_active":"2023-10-03T18:05:46.982469+0000","last_peered":"2023-10-03T18:05:46.982469+0000","last_clean":"2023-10-03T18:05:46.982469+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:05:46.982469+0000","last_undegraded":"2023-10-03T18:05:46.982469+0000","last_fullsized":"2023-10-03T18:05:46.982469+0000","mapping_epoch":12,"log_start":"93'119945","ondisk_log_start":"93'119945","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":282,"snaptrim_duration":36.697120716999997,"stat_sum":{"num_bytes":2953294666,"num_objects":3592,"num_object_clones":2893,"num_object_copies":7184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3592,"num_whiteouts":0,"num_read":127699,"num_read_kb":484751,"num_write":122740,"num_write_kb":490702,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"a"}]},{"pgid":"2.1","version":"105'118153","reported_seq":229826,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:49.742405+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:05:49.742405+0000","last_peered":"2023-10-03T18:05:49.742405+0000","last_clean":"2023-10-03T18:05:49.742405+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:05:49.742405+0000","last_undegraded":"2023-10-03T18:05:49.742405+0000","last_fullsized":"2023-10-03T18:05:49.742405+0000","mapping_epoch":12,"log_start":"91'108130","ondisk_log_start":"91'108130","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1424,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 251s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2691687021,"num_objects":3663,"num_object_clones":3028,"num_object_copies":7326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3663,"num_whiteouts":0,"num_read":117079,"num_read_kb":439487,"num_write":112026,"num_write_kb":447813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"105'118418","reported_seq":232828,"reported_epoch":105,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:05:46.760463+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:05:46.760463+0000","last_peered":"2023-10-03T18:05:46.760463+0000","last_clean":"2023-10-03T18:05:46.760463+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:05:46.760463+0000","last_undegraded":"2023-10-03T18:05:46.760463+0000","last_fullsized":"2023-10-03T18:05:46.760463+0000","mapping_epoch":12,"log_start":"91'108335","ondisk_log_start":"91'108335","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1715,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 253s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2693989126,"num_objects":3657,"num_object_clones":3015,"num_object_copies":7314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3657,"num_whiteouts":0,"num_read":121037,"num_read_kb":445581,"num_write":112785,"num_write_kb":450665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"105'118294","reported_seq":230132,"reported_epoch":105,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:05:47.012011+0000","last_change":"2023-10-03T18:05:46.561332+0000","last_active":"2023-10-03T18:05:47.012011+0000","last_peered":"2023-10-03T18:05:47.012011+0000","last_clean":"2023-10-03T18:05:47.012011+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:05:47.012011+0000","last_undegraded":"2023-10-03T18:05:47.012011+0000","last_fullsized":"2023-10-03T18:05:47.012011+0000","mapping_epoch":12,"log_start":"91'108240","ondisk_log_start":"91'108240","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":274,"snaptrim_duration":38.020810394999998,"stat_sum":{"num_bytes":2681696995,"num_objects":3222,"num_object_clones":2584,"num_object_copies":6444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3222,"num_whiteouts":0,"num_read":116657,"num_read_kb":439263,"num_write":111433,"num_write_kb":445450,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22270960913,"num_objects":27848,"num_object_clones":22584,"num_object_copies":55696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27848,"num_whiteouts":0,"num_read":971725,"num_read_kb":3654612,"num_write":927578,"num_write_kb":3707909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7592673280,"data_stored":37886769698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38147,"internal_metadata":0},"log_size":80451,"ondisk_log_size":80451,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738766,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3116420,"kb_used_data":1424956,"kb_used_omap":1,"kb_used_meta":1691454,"kb_avail":91255420,"statfs":{"total":96636764160,"available":93445550080,"internally_reserved":0,"allocated":1459154944,"data_stored":7092857925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1732049350},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.1309999999999998,"5min":2.2440000000000002,"15min":2.5880000000000001},"min":{"1min":0.186,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.9450000000000003,"5min":10.145,"15min":22.562000000000001},"last":0.22},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.4980000000000002,"15min":2.8500000000000001},"min":{"1min":0.21199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.423999999999999,"5min":14.52,"15min":21.827999999999999},"last":10.423999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.9930000000000001,"15min":2.4380000000000002},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.945,"5min":13.945,"15min":26.074000000000002},"last":10.468999999999999},{"interface":"front","average":{"1min":2.7109999999999999,"5min":2.1259999999999999,"15min":2},"min":{"1min":0.245,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":13.885,"5min":13.885,"15min":16.457000000000001},"last":1.258}]}]},{"osd":0,"up_from":8,"seq":34359738765,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3104164,"kb_used_data":2284700,"kb_used_omap":347,"kb_used_meta":819108,"kb_avail":91267676,"statfs":{"total":96636764160,"available":93458100224,"internally_reserved":0,"allocated":2339532800,"data_stored":11853996778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":838766768},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":4.3120000000000003,"5min":4.1219999999999999,"15min":3.9660000000000002},"min":{"1min":0.221,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":10.057,"5min":30.852,"15min":30.852},"last":2.04},{"interface":"front","average":{"1min":4.3040000000000003,"5min":3.1200000000000001,"15min":3.621},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.19400000000000001},"max":{"1min":9.7620000000000005,"5min":16.584,"15min":24.215},"last":1.919}]},{"osd":2,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.4670000000000001,"15min":2.452},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":8.2899999999999991,"5min":14.195,"15min":14.195},"last":0.52100000000000002},{"interface":"front","average":{"1min":2.855,"5min":2.3090000000000002,"15min":2.3919999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":10.949,"5min":11.94,"15min":13.677},"last":0.33500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738758,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4577916,"kb_used_data":3707956,"kb_used_omap":1915,"kb_used_meta":868036,"kb_avail":89793924,"statfs":{"total":96636764160,"available":91948978176,"internally_reserved":0,"allocated":3796946944,"data_stored":18937097008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":888869809},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,10,21,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.9649999999999999,"15min":3.5110000000000001},"min":{"1min":0.188,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.093999999999999,"5min":12.513999999999999,"15min":39.990000000000002},"last":0.91100000000000003},{"interface":"front","average":{"1min":3.6850000000000001,"5min":4.1909999999999998,"15min":3.952},"min":{"1min":0.26500000000000001,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":20.033000000000001,"5min":20.716999999999999,"15min":37.381},"last":0.93600000000000005}]},{"osd":2,"last update":"Tue Oct 3 18:04:48 2023","interfaces":[{"interface":"back","average":{"1min":2.0550000000000002,"5min":3.3860000000000001,"15min":2.4529999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":13.840999999999999,"5min":24.579000000000001,"15min":24.579000000000001},"last":9.9209999999999994},{"interface":"front","average":{"1min":3.702,"5min":2.6030000000000002,"15min":3.2909999999999999},"min":{"1min":0.23400000000000001,"5min":0.17299999999999999,"15min":0.17299999999999999},"max":{"1min":20.108000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2338369536,"data_stored":11850489035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3795697664,"data_stored":18938635537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1458606080,"data_stored":7097645126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:05:51.223 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.223 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.223 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.223 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.224 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.224 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.224 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.224 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:05:51.224 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:05:51.397 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:51.397+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:51.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:51.483+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:51.504 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:51.497+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:51.504 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:51.497+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:52.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.436+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:52.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.491+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:52.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.519+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:52.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.534+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:52.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.550+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:52.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.562+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:52.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.581+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:52.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.581+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:52.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.614+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:52.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.614+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:52.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:52.624+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:54.580 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:54.575+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:05:54.594 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:54.592+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:05:54.595 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:05:54.595+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:05:55.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.898+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:55.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.898+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:55.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.898+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:55.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.916+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:55.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.916+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:55.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:55.916+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:56.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.228+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:56.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.243+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:56.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.243+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:56.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.783+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:56.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.838+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:56.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.838+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:56.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.850+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:56.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.913+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:56.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.913+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:56.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.913+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:56.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.926+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:56.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:56.928+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:57.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.676+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:05:57.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.712+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:05:57.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.712+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:57.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.739+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:57.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.740+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:57.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.781+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:57.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.803+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:57.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.803+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:57.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.810+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:57.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.810+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:57.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:57.830+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:58.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.413+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:05:58.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.413+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:05:58.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.413+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:05:58.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.539+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:58.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.539+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:58.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.539+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:58.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:58.539+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:05:59.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.795+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:05:59.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.878+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:05:59.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.878+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:05:59.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.883+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:05:59.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.937+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:05:59.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:05:59.937+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:02.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.595+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:02.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.650+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:02.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.853+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:02.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.853+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:02.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.853+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:02.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.871+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:02.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.934+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:02.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.934+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:02.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.940+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:02.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:02.961+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:03.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.021+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:03.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.344+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:03.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.344+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:03.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.345+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:03.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.365+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:03.411 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.409+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:03.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.428+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:03.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.615+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:03.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.615+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:03.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.618+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:03.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.799+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:03.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:03.799+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:03.984 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:03.984+0000 7f9f21f20700 -1 reset not still connected to 0x5632f0325110 2023-10-03T18:06:04.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.020+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:04.243 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:04.243+0000 7f9f21f20700 -1 reset not still connected to 0x5632e844f6c0 2023-10-03T18:06:04.243 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:04.243+0000 7f9f21f20700 -1 reset not still connected to 0x5632f0325110 2023-10-03T18:06:04.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.244+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:04.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.428+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:04.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.428+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:04.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.428+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:04.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.428+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:04.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.434+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:04.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.434+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:04.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.447+0000 7fc515f98700 -1 reset not still connected to 0x56220630ab60 2023-10-03T18:06:04.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.480+0000 7fc515f98700 -1 reset not still connected to 0x562209070b60 2023-10-03T18:06:04.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.513+0000 7fc515f98700 -1 reset not still connected to 0x56220af2a000 2023-10-03T18:06:04.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.513+0000 7fc515f98700 -1 reset not still connected to 0x5622113075f0 2023-10-03T18:06:04.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.540+0000 7fc515f98700 -1 reset not still connected to 0x562222c8eb60 2023-10-03T18:06:04.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.825+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:04.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.825+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:04.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:04.825+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:05.050 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:05.049+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:05.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:05.092+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:11.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.116+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:11.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.171+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:11.226 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:06:11.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:11.272 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:11.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.318+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:11.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.330+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:11.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.331+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:11.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.331+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:11.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.375+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:11.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.459+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:11.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:11.492+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:11.868 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:06:11.899 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:06:11.918 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1106,"stamp":"2023-10-03T18:06:10.596357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22297327409,"num_objects":28491,"num_object_clones":23225,"num_object_copies":56982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28491,"num_whiteouts":0,"num_read":981416,"num_read_kb":3683237,"num_write":934199,"num_write_kb":3735019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80555,"ondisk_log_size":80555,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11011012,"kb_used_data":7467688,"kb_used_omap":2264,"kb_used_meta":3541031,"kb_avail":272104508,"statfs":{"total":289910292480,"available":278635016192,"internally_reserved":0,"allocated":7646912512,"data_stored":38787287119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":3626016295},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,5,7,16,8,16,21,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6393704,"num_objects":737,"num_object_clones":740,"num_object_copies":1474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":3176,"num_read_kb":7662,"num_write":1685,"num_write_kb":6299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.003635"},"pg_stats":[{"pgid":"2.7","version":"112'123827","reported_seq":239786,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:08.778157+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:06:08.778157+0000","last_peered":"2023-10-03T18:06:08.778157+0000","last_clean":"2023-10-03T18:06:08.778157+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:06:08.778157+0000","last_undegraded":"2023-10-03T18:06:08.778157+0000","last_fullsized":"2023-10-03T18:06:08.778157+0000","mapping_epoch":12,"log_start":"93'113733","ondisk_log_start":"93'113733","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1532,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2779242980,"num_objects":3937,"num_object_clones":3286,"num_object_copies":7874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3937,"num_whiteouts":0,"num_read":120938,"num_read_kb":459004,"num_write":116344,"num_write_kb":465089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'113006","reported_seq":220584,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:08.721527+0000","last_change":"2023-10-03T18:06:08.339361+0000","last_active":"2023-10-03T18:06:08.721527+0000","last_peered":"2023-10-03T18:06:08.721527+0000","last_clean":"2023-10-03T18:06:08.721527+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:06:08.721527+0000","last_undegraded":"2023-10-03T18:06:08.721527+0000","last_fullsized":"2023-10-03T18:06:08.721527+0000","mapping_epoch":12,"log_start":"93'102949","ondisk_log_start":"93'102949","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.44420918599999998,"stat_sum":{"num_bytes":2566607674,"num_objects":3104,"num_object_clones":2496,"num_object_copies":6208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3104,"num_whiteouts":0,"num_read":113192,"num_read_kb":424888,"num_write":107330,"num_write_kb":428963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"4"}]},{"pgid":"2.5","version":"112'136821","reported_seq":262926,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:08.728335+0000","last_change":"2023-10-03T18:06:08.680750+0000","last_active":"2023-10-03T18:06:08.728335+0000","last_peered":"2023-10-03T18:06:08.728335+0000","last_clean":"2023-10-03T18:06:08.728335+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:06:08.728335+0000","last_undegraded":"2023-10-03T18:06:08.728335+0000","last_fullsized":"2023-10-03T18:06:08.728335+0000","mapping_epoch":12,"log_start":"93'126748","ondisk_log_start":"93'126748","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":26,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":1.9652858580000001,"stat_sum":{"num_bytes":3009053124,"num_objects":3664,"num_object_clones":2958,"num_object_copies":7328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":133746,"num_read_kb":497243,"num_write":125985,"num_write_kb":503455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"2"}]},{"pgid":"2.0","version":"112'128859","reported_seq":249420,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:09.842561+0000","last_change":"2023-10-03T18:06:07.331434+0000","last_active":"2023-10-03T18:06:09.842561+0000","last_peered":"2023-10-03T18:06:09.842561+0000","last_clean":"2023-10-03T18:06:09.842561+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:06:09.842561+0000","last_undegraded":"2023-10-03T18:06:09.842561+0000","last_fullsized":"2023-10-03T18:06:09.842561+0000","mapping_epoch":12,"log_start":"93'118845","ondisk_log_start":"93'118845","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":0.68257931599999999,"stat_sum":{"num_bytes":2908488436,"num_objects":3265,"num_object_clones":2578,"num_object_copies":6530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":126113,"num_read_kb":478889,"num_write":122227,"num_write_kb":488610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"b"}]},{"pgid":"1.0","version":"88'99","reported_seq":266,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-03T18:06:05.668003+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:06:05.668003+0000","last_peered":"2023-10-03T18:06:05.668003+0000","last_clean":"2023-10-03T18:06:05.668003+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:06:05.668003+0000","last_undegraded":"2023-10-03T18:06:05.668003+0000","last_fullsized":"2023-10-03T18:06:05.668003+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'131349","reported_seq":254338,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:08.686872+0000","last_change":"2023-10-03T18:06:08.686919+0000","last_active":"2023-10-03T18:06:08.686872+0000","last_peered":"2023-10-03T18:06:08.686872+0000","last_clean":"2023-10-03T18:06:08.686872+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:06:08.686872+0000","last_undegraded":"2023-10-03T18:06:08.686872+0000","last_fullsized":"2023-10-03T18:06:08.686872+0000","mapping_epoch":12,"log_start":"93'121249","ondisk_log_start":"93'121249","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.65435029099999997,"stat_sum":{"num_bytes":2956686104,"num_objects":3615,"num_object_clones":2917,"num_object_copies":7230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3615,"num_whiteouts":0,"num_read":128887,"num_read_kb":488576,"num_write":123588,"num_write_kb":494018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"3"}]},{"pgid":"2.1","version":"112'119241","reported_seq":231542,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:09.846336+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:06:09.846336+0000","last_peered":"2023-10-03T18:06:09.846336+0000","last_clean":"2023-10-03T18:06:09.846336+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:06:09.846336+0000","last_undegraded":"2023-10-03T18:06:09.846336+0000","last_fullsized":"2023-10-03T18:06:09.846336+0000","mapping_epoch":12,"log_start":"93'109233","ondisk_log_start":"93'109233","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1638,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 271s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2694464111,"num_objects":3851,"num_object_clones":3216,"num_object_copies":7702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3851,"num_whiteouts":0,"num_read":118186,"num_read_kb":442535,"num_write":112733,"num_write_kb":450533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"112'119495","reported_seq":234763,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:06.913058+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:06:06.913058+0000","last_peered":"2023-10-03T18:06:06.913058+0000","last_clean":"2023-10-03T18:06:06.913058+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:06:06.913058+0000","last_undegraded":"2023-10-03T18:06:06.913058+0000","last_fullsized":"2023-10-03T18:06:06.913058+0000","mapping_epoch":12,"log_start":"93'109438","ondisk_log_start":"93'109438","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1860,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 274s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2697130857,"num_objects":3807,"num_object_clones":3163,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3807,"num_whiteouts":0,"num_read":122436,"num_read_kb":449183,"num_write":113588,"num_write_kb":453741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"112'119700","reported_seq":232237,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:08.709125+0000","last_change":"2023-10-03T18:06:08.320916+0000","last_active":"2023-10-03T18:06:08.709125+0000","last_peered":"2023-10-03T18:06:08.709125+0000","last_clean":"2023-10-03T18:06:08.709125+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:06:08.709125+0000","last_undegraded":"2023-10-03T18:06:08.709125+0000","last_fullsized":"2023-10-03T18:06:08.709125+0000","mapping_epoch":12,"log_start":"93'109647","ondisk_log_start":"93'109647","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":19,"snaptrim_duration":1.645376014,"stat_sum":{"num_bytes":2685063755,"num_objects":3246,"num_object_clones":2611,"num_object_copies":6492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3246,"num_whiteouts":0,"num_read":117812,"num_read_kb":442828,"num_write":112267,"num_write_kb":448716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22296737041,"num_objects":28489,"num_object_clones":23225,"num_object_copies":56978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28489,"num_whiteouts":0,"num_read":981310,"num_read_kb":3683146,"num_write":934062,"num_write_kb":3733125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7643901952,"data_stored":38883392034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38147,"internal_metadata":0},"log_size":80456,"ondisk_log_size":80456,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738770,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3135016,"kb_used_data":1433824,"kb_used_omap":1,"kb_used_meta":1701182,"kb_avail":91236824,"statfs":{"total":96636764160,"available":93426507776,"internally_reserved":0,"allocated":1468235776,"data_stored":7312268432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1742010822},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1,3,8,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.1309999999999998,"5min":2.2440000000000002,"15min":2.5880000000000001},"min":{"1min":0.186,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.9450000000000003,"5min":10.145,"15min":22.562000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.4980000000000002,"15min":2.8500000000000001},"min":{"1min":0.21199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.423999999999999,"5min":14.52,"15min":21.827999999999999},"last":1.097}]},{"osd":1,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.9930000000000001,"15min":2.4380000000000002},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.945,"5min":13.945,"15min":26.074000000000002},"last":5.6310000000000002},{"interface":"front","average":{"1min":2.7109999999999999,"5min":2.1259999999999999,"15min":2},"min":{"1min":0.245,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":13.885,"5min":13.885,"15min":16.457000000000001},"last":0.69899999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738769,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3204364,"kb_used_data":2300228,"kb_used_omap":347,"kb_used_meta":903780,"kb_avail":91167476,"statfs":{"total":96636764160,"available":93355495424,"internally_reserved":0,"allocated":2355433472,"data_stored":12048855161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":925470896},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":4.3120000000000003,"5min":4.1219999999999999,"15min":3.9660000000000002},"min":{"1min":0.221,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":10.057,"5min":30.852,"15min":30.852},"last":0.55000000000000004},{"interface":"front","average":{"1min":4.3040000000000003,"5min":3.1200000000000001,"15min":3.621},"min":{"1min":0.29099999999999998,"5min":0.219,"15min":0.19400000000000001},"max":{"1min":9.7620000000000005,"5min":16.584,"15min":24.215},"last":0.317}]},{"osd":2,"last update":"Tue Oct 3 18:05:02 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.4670000000000001,"15min":2.452},"min":{"1min":0.26600000000000001,"5min":0.216,"15min":0.20100000000000001},"max":{"1min":8.2899999999999991,"5min":14.195,"15min":14.195},"last":0.26300000000000001},{"interface":"front","average":{"1min":2.855,"5min":2.3090000000000002,"15min":2.3919999999999999},"min":{"1min":0.255,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":10.949,"5min":11.94,"15min":13.677},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738762,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4671632,"kb_used_data":3733636,"kb_used_omap":1915,"kb_used_meta":936068,"kb_avail":89700208,"statfs":{"total":96636764160,"available":91853012992,"internally_reserved":0,"allocated":3823243264,"data_stored":19426163526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":958534577},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,4,6,13,7,11,13,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:05:50 2023","interfaces":[{"interface":"back","average":{"1min":3.556,"5min":2.7949999999999999,"15min":3.367},"min":{"1min":0.25800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.997,"5min":16.997,"15min":16.997},"last":0.223},{"interface":"front","average":{"1min":2.4630000000000001,"5min":3.4350000000000001,"15min":3.7919999999999998},"min":{"1min":0.27500000000000002,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":16.405000000000001,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.88}]},{"osd":2,"last update":"Tue Oct 3 18:05:50 2023","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.0600000000000001,"15min":2.5699999999999998},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":11.866,"5min":17.492999999999999,"15min":24.579000000000001},"last":1.3120000000000001},{"interface":"front","average":{"1min":2.1709999999999998,"5min":2.6629999999999998,"15min":3.1139999999999999},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.17299999999999999},"max":{"1min":10.622,"5min":20.108000000000001,"15min":22.739000000000001},"last":2.4710000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2354053120,"data_stored":12061441174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3822223360,"data_stored":19477951761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1467625472,"data_stored":7343999099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:06:11.920 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.920 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.920 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.920 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.920 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.921 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.921 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.921 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:11.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.7 2023-10-03T18:06:12.653 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.7 on osd.1 to deep-scrub 2023-10-03T18:06:12.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:06:13.378 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:06:13.391 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:06:14.083 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:06:14.104 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:06:14.239 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.239+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:14.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.290+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:14.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.382+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:14.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.382+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:14.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.382+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:14.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.391+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:14.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.447+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:14.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.447+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:14.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.448+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:14.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.464+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:14.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:14.620+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:14.804 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:06:14.820 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:06:15.518 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:06:15.544 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.1 2023-10-03T18:06:16.234 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.1 on osd.2 to deep-scrub 2023-10-03T18:06:16.253 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.2 2023-10-03T18:06:16.880 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.2 on osd.0 to deep-scrub 2023-10-03T18:06:16.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:06:17.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.191+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:17.254 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.253+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:17.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.266+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:17.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.348+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:17.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.355+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:17.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.355+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:17.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.360+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:17.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.424+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:17.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.424+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:17.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.468+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:17.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.468+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:17.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.598+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:17.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.614+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:17.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.614+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:17.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.614+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:17.627 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:06:17.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.635+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:17.642 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:06:17.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.674+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:17.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:17.674+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:22.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.131+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:22.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.131+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:22.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.217+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:22.264 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.262+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:22.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.264+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:22.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.264+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:22.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.292+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:22.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.332+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:22.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:22.332+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:25.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.554+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:25.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.554+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:25.692 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.689+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:25.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.718+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:25.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.718+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:25.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.737+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:25.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.881+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:25.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.885+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:25.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.887+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:25.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.916+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:25.923 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.916+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:25.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:25.994+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:26.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.025+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:26.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.025+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:26.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.025+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:26.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.059+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:26.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.137+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:26.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:26.137+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:27.473 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.471+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:27.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.622+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:27.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.622+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:27.647 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.646+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:27.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.665+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:27.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.671+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:27.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.717+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:27.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.717+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:27.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.727+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:27.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.730+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:27.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:27.735+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:32.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.092+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:32.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.117+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:32.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.189+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:32.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.190+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:32.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.211+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:32.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.286+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:32.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:32.286+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:33.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.045+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:33.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.049+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:33.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.136+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:33.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.420+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:33.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.505+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:33.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.505+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:33.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.523+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:33.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.523+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:33.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:33.529+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:34.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.311+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:34.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.397+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:34.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.397+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:34.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.397+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:34.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.397+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:34.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.398+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:34.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.398+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:34.405 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.399+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:34.411 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.405+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:34.411 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:34.405+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:37.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:06:38.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.108+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:38.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.156+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:38.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.235+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:38.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.319+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:38.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.319+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:38.371 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:06:38.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.373+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:38.376 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:06:38.399 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1119,"stamp":"2023-10-03T18:06:36.606568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22325090097,"num_objects":30271,"num_object_clones":25005,"num_object_copies":60542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30271,"num_whiteouts":0,"num_read":988114,"num_read_kb":3709885,"num_write":940922,"num_write_kb":3761911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80272,"ondisk_log_size":80272,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11473116,"kb_used_data":7521020,"kb_used_omap":2264,"kb_used_meta":3949799,"kb_avail":271642404,"statfs":{"total":289910292480,"available":278161821696,"internally_reserved":0,"allocated":7701524480,"data_stored":41717988001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":4044594727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4,3,15,19,20,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17252352,"num_objects":627,"num_object_clones":627,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":0,"num_read":4153,"num_read_kb":16512,"num_write":4172,"num_write_kb":16688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005924"},"pg_stats":[{"pgid":"2.7","version":"112'125351","reported_seq":241793,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:36.194348+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:06:36.194348+0000","last_peered":"2023-10-03T18:06:36.194348+0000","last_clean":"2023-10-03T18:06:36.194348+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:06:36.194348+0000","last_undegraded":"2023-10-03T18:06:36.194348+0000","last_fullsized":"2023-10-03T18:06:36.194348+0000","mapping_epoch":12,"log_start":"93'115340","ondisk_log_start":"93'115340","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1809,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 298s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2782945764,"num_objects":4309,"num_object_clones":3658,"num_object_copies":8618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4309,"num_whiteouts":0,"num_read":121773,"num_read_kb":462340,"num_write":117242,"num_write_kb":468681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'114458","reported_seq":222451,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:36.160520+0000","last_change":"2023-10-03T18:06:35.086358+0000","last_active":"2023-10-03T18:06:36.160520+0000","last_peered":"2023-10-03T18:06:36.160520+0000","last_clean":"2023-10-03T18:06:36.160520+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:06:36.160520+0000","last_undegraded":"2023-10-03T18:06:36.160520+0000","last_fullsized":"2023-10-03T18:06:36.160520+0000","mapping_epoch":12,"log_start":"93'104456","ondisk_log_start":"93'104456","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":183,"snaptrim_duration":25.919287218000001,"stat_sum":{"num_bytes":2569679674,"num_objects":3273,"num_object_clones":2665,"num_object_copies":6546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":114004,"num_read_kb":428112,"num_write":108081,"num_write_kb":431967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"a"}]},{"pgid":"2.5","version":"112'138469","reported_seq":265051,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:36.060930+0000","last_change":"2023-10-03T18:06:35.184556+0000","last_active":"2023-10-03T18:06:36.060930+0000","last_peered":"2023-10-03T18:06:36.060930+0000","last_clean":"2023-10-03T18:06:36.060930+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:06:36.060930+0000","last_undegraded":"2023-10-03T18:06:36.060930+0000","last_fullsized":"2023-10-03T18:06:36.060930+0000","mapping_epoch":12,"log_start":"96'128452","ondisk_log_start":"96'128452","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":182,"snaptrim_duration":28.469108386999999,"stat_sum":{"num_bytes":3012764100,"num_objects":3896,"num_object_clones":3190,"num_object_copies":7792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3896,"num_whiteouts":0,"num_read":134656,"num_read_kb":500863,"num_write":126881,"num_write_kb":507039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"7"}]},{"pgid":"2.0","version":"112'130872","reported_seq":251814,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:35.117116+0000","last_change":"2023-10-03T18:06:33.184417+0000","last_active":"2023-10-03T18:06:35.117116+0000","last_peered":"2023-10-03T18:06:35.117116+0000","last_clean":"2023-10-03T18:06:35.117116+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:06:35.117116+0000","last_undegraded":"2023-10-03T18:06:35.117116+0000","last_fullsized":"2023-10-03T18:06:35.117116+0000","mapping_epoch":12,"log_start":"97'120860","ondisk_log_start":"97'120860","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":442,"snaptrim_duration":26.535597594999999,"stat_sum":{"num_bytes":2912031476,"num_objects":3216,"num_object_clones":2529,"num_object_copies":6432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3216,"num_whiteouts":0,"num_read":126983,"num_read_kb":482361,"num_write":123076,"num_write_kb":492006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":266,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-03T18:06:05.668003+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:06:05.668003+0000","last_peered":"2023-10-03T18:06:05.668003+0000","last_clean":"2023-10-03T18:06:05.668003+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:06:05.668003+0000","last_undegraded":"2023-10-03T18:06:05.668003+0000","last_fullsized":"2023-10-03T18:06:05.668003+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'133094","reported_seq":256513,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:36.121850+0000","last_change":"2023-10-03T18:06:35.327039+0000","last_active":"2023-10-03T18:06:36.121850+0000","last_peered":"2023-10-03T18:06:36.121850+0000","last_clean":"2023-10-03T18:06:36.121850+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:06:36.121850+0000","last_undegraded":"2023-10-03T18:06:36.121850+0000","last_fullsized":"2023-10-03T18:06:36.121850+0000","mapping_epoch":12,"log_start":"94'123059","ondisk_log_start":"94'123059","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":198,"snaptrim_duration":26.656260495000001,"stat_sum":{"num_bytes":2960593688,"num_objects":3809,"num_object_clones":3111,"num_object_copies":7618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3809,"num_whiteouts":0,"num_read":129780,"num_read_kb":492120,"num_write":124523,"num_write_kb":497758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"8"}]},{"pgid":"2.1","version":"112'120645","reported_seq":233403,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:35.109690+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:06:35.109690+0000","last_peered":"2023-10-03T18:06:35.109690+0000","last_clean":"2023-10-03T18:06:35.109690+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:06:35.109690+0000","last_undegraded":"2023-10-03T18:06:35.109690+0000","last_fullsized":"2023-10-03T18:06:35.109690+0000","mapping_epoch":12,"log_start":"93'110637","ondisk_log_start":"93'110637","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1878,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 296s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2697962095,"num_objects":4196,"num_object_clones":3561,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":118950,"num_read_kb":445563,"num_write":113581,"num_write_kb":453925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"112'120817","reported_seq":236540,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:32.050367+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:06:32.050367+0000","last_peered":"2023-10-03T18:06:32.050367+0000","last_clean":"2023-10-03T18:06:32.050367+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:06:32.050367+0000","last_undegraded":"2023-10-03T18:06:32.050367+0000","last_fullsized":"2023-10-03T18:06:32.050367+0000","mapping_epoch":12,"log_start":"93'110743","ondisk_log_start":"93'110743","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2138,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 299s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2700116841,"num_objects":4159,"num_object_clones":3515,"num_object_copies":8318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4159,"num_whiteouts":0,"num_read":123206,"num_read_kb":452247,"num_write":114317,"num_write_kb":456657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"112'121270","reported_seq":234241,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:36.097956+0000","last_change":"2023-10-03T18:06:35.084724+0000","last_active":"2023-10-03T18:06:36.097956+0000","last_peered":"2023-10-03T18:06:36.097956+0000","last_clean":"2023-10-03T18:06:36.097956+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:06:36.097956+0000","last_undegraded":"2023-10-03T18:06:36.097956+0000","last_fullsized":"2023-10-03T18:06:36.097956+0000","mapping_epoch":12,"log_start":"93'111256","ondisk_log_start":"93'111256","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":207,"snaptrim_duration":28.409188455999999,"stat_sum":{"num_bytes":2688406091,"num_objects":3411,"num_object_clones":2776,"num_object_copies":6822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3411,"num_whiteouts":0,"num_read":118656,"num_read_kb":446188,"num_write":113084,"num_write_kb":451984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22324499729,"num_objects":30269,"num_object_clones":25005,"num_object_copies":60538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30269,"num_whiteouts":0,"num_read":988008,"num_read_kb":3709794,"num_write":940785,"num_write_kb":3760017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7698440192,"data_stored":41719855650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38147,"internal_metadata":0},"log_size":80173,"ondisk_log_size":80173,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738775,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3231668,"kb_used_data":1444396,"kb_used_omap":1,"kb_used_meta":1787262,"kb_avail":91140172,"statfs":{"total":96636764160,"available":93327536128,"internally_reserved":0,"allocated":1479061504,"data_stored":7720717456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1830156742},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.1309999999999998,"5min":2.2440000000000002,"15min":2.5880000000000001},"min":{"1min":0.186,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.9450000000000003,"5min":10.145,"15min":22.562000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.4980000000000002,"15min":2.8500000000000001},"min":{"1min":0.21199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":10.423999999999999,"5min":14.52,"15min":21.827999999999999},"last":0.28499999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:05:46 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":1.9930000000000001,"15min":2.4380000000000002},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.945,"5min":13.945,"15min":26.074000000000002},"last":2.306},{"interface":"front","average":{"1min":2.7109999999999999,"5min":2.1259999999999999,"15min":2},"min":{"1min":0.245,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":13.885,"5min":13.885,"15min":16.457000000000001},"last":0.80200000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738774,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3287336,"kb_used_data":2315232,"kb_used_omap":347,"kb_used_meta":971748,"kb_avail":91084504,"statfs":{"total":96636764160,"available":93270532096,"internally_reserved":0,"allocated":2370797568,"data_stored":13114024651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":995070128},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:06:07 2023","interfaces":[{"interface":"back","average":{"1min":1.9419999999999999,"5min":3.8839999999999999,"15min":3.8540000000000001},"min":{"1min":0.17799999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":8.9849999999999994,"5min":30.852,"15min":30.852},"last":2.5249999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":2.8300000000000001,"15min":3.4470000000000001},"min":{"1min":0.29399999999999998,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":5.4969999999999999,"5min":16.584,"15min":24.215},"last":2.5659999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:06:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":2.3079999999999998,"15min":2.2610000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.4779999999999998,"5min":14.195,"15min":14.195},"last":1.97},{"interface":"front","average":{"1min":1.464,"5min":2.1160000000000001,"15min":2.3100000000000001},"min":{"1min":0.26600000000000001,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":5.4889999999999999,"5min":10.949,"15min":13.677},"last":0.27600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738767,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4954112,"kb_used_data":3761392,"kb_used_omap":1915,"kb_used_meta":1190788,"kb_avail":89417728,"statfs":{"total":96636764160,"available":91563753472,"internally_reserved":0,"allocated":3851665408,"data_stored":20883245894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1219367857},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,3,9,14,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:05:50 2023","interfaces":[{"interface":"back","average":{"1min":3.556,"5min":2.7949999999999999,"15min":3.367},"min":{"1min":0.25800000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":16.997,"5min":16.997,"15min":16.997},"last":6.0880000000000001},{"interface":"front","average":{"1min":2.4630000000000001,"5min":3.4350000000000001,"15min":3.7919999999999998},"min":{"1min":0.27500000000000002,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":16.405000000000001,"5min":20.033000000000001,"15min":24.481000000000002},"last":6.7910000000000004}]},{"osd":2,"last update":"Tue Oct 3 18:05:50 2023","interfaces":[{"interface":"back","average":{"1min":3.29,"5min":3.0600000000000001,"15min":2.5699999999999998},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":11.866,"5min":17.492999999999999,"15min":24.579000000000001},"last":6.2400000000000002},{"interface":"front","average":{"1min":2.1709999999999998,"5min":2.6629999999999998,"15min":3.1139999999999999},"min":{"1min":0.22500000000000001,"5min":0.215,"15min":0.17299999999999999},"max":{"1min":10.622,"5min":20.108000000000001,"15min":22.739000000000001},"last":6.5220000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2369515520,"data_stored":13119859862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3850792960,"data_stored":20879688977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1478131712,"data_stored":7720306811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:06:38.400 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.401 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.401 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.401 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.401 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.401 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.402 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.402 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:38.402 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:06:38.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.408+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:38.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.906+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:38.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.935+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:38.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.935+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:38.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.935+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:38.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:38.935+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:39.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.015+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:39.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.015+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:39.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.236+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:39.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.296+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:39.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.307+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:39.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.307+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:39.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.382+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:39.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.461+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:39.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:39.461+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:40.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.844+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:40.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.844+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:40.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.878+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:40.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.914+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:40.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.926+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:40.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.926+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:40.946 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.946+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:40.962 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.962+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:41.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:40.993+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:44.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.599+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:44.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.623+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:44.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.623+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:44.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.700+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:44.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.704+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:44.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.720+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:44.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.720+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:44.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.755+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:44.771 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.771+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:44.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:44.808+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:47.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.474+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:06:47.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.553+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:47.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.577+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:47.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.590+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:47.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.590+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:47.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.597+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:47.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.612+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:47.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.650+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:47.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.650+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:47.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.650+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:47.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:47.650+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:48.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.280+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:06:48.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.285+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:48.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.299+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:48.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.314+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:48.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.314+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:48.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.347+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:48.410 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:48.410+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:50.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.280+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:50.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.306+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:50.313 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.312+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:50.543 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.542+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:50.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.557+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:50.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.557+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:50.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.564+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:50.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.591+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:50.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:50.597+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:51.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:51.338+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:51.509 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:51.508+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:51.509 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:51.508+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:51.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:51.511+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:51.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:51.511+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:53.405 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.397+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:53.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.431+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:53.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.444+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:53.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.445+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:53.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.484+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:53.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:53.492+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:54.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:54.103+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:54.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:54.103+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:54.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:54.190+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:54.329 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:54.320+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:54.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:54.339+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:57.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.498+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:06:57.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.498+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:06:57.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.506+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:06:57.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.506+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:06:57.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.513+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:06:57.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.513+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:06:57.517 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.516+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:06:57.517 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.516+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:06:57.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:06:57.523+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:06:58.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:06:59.151 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:06:59.186 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:06:59.207 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1130,"stamp":"2023-10-03T18:06:58.617508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22354290481,"num_objects":30244,"num_object_clones":24978,"num_object_copies":60488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30244,"num_whiteouts":0,"num_read":996280,"num_read_kb":3737780,"num_write":947934,"num_write_kb":3789959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80510,"ondisk_log_size":80510,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11589616,"kb_used_data":7578200,"kb_used_omap":2264,"kb_used_meta":4009127,"kb_avail":271525904,"statfs":{"total":289910292480,"available":278042525696,"internally_reserved":0,"allocated":7760076800,"data_stored":41782442657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":4105346599},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,12,16,23,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20439040,"num_objects":-109,"num_object_clones":-109,"num_object_copies":-218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-109,"num_whiteouts":0,"num_read":6167,"num_read_kb":19943,"num_write":4891,"num_write_kb":19564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007442"},"pg_stats":[{"pgid":"2.7","version":"112'126492","reported_seq":243766,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:57.247123+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:06:57.247123+0000","last_peered":"2023-10-03T18:06:57.247123+0000","last_clean":"2023-10-03T18:06:57.247123+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:06:57.247123+0000","last_undegraded":"2023-10-03T18:06:57.247123+0000","last_fullsized":"2023-10-03T18:06:57.247123+0000","mapping_epoch":12,"log_start":"93'116441","ondisk_log_start":"93'116441","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1983,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 319s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2786550244,"num_objects":4403,"num_object_clones":3752,"num_object_copies":8806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4403,"num_whiteouts":0,"num_read":122787,"num_read_kb":465964,"num_write":118112,"num_write_kb":472161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"112'115721","reported_seq":224373,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:57.261083+0000","last_change":"2023-10-03T18:06:56.588008+0000","last_active":"2023-10-03T18:06:57.261083+0000","last_peered":"2023-10-03T18:06:57.261083+0000","last_clean":"2023-10-03T18:06:57.261083+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:06:57.261083+0000","last_undegraded":"2023-10-03T18:06:57.261083+0000","last_fullsized":"2023-10-03T18:06:57.261083+0000","mapping_epoch":12,"log_start":"93'105660","ondisk_log_start":"93'105660","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":365,"snaptrim_duration":49.931629592,"stat_sum":{"num_bytes":2573087546,"num_objects":3176,"num_object_clones":2568,"num_object_copies":6352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3176,"num_whiteouts":0,"num_read":114906,"num_read_kb":431237,"num_write":108880,"num_write_kb":435163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"11"}]},{"pgid":"2.5","version":"112'139957","reported_seq":267280,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:57.241828+0000","last_change":"2023-10-03T18:06:55.762071+0000","last_active":"2023-10-03T18:06:57.241828+0000","last_peered":"2023-10-03T18:06:57.241828+0000","last_clean":"2023-10-03T18:06:57.241828+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:06:57.241828+0000","last_undegraded":"2023-10-03T18:06:57.241828+0000","last_fullsized":"2023-10-03T18:06:57.241828+0000","mapping_epoch":12,"log_start":"97'129956","ondisk_log_start":"97'129956","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":374,"snaptrim_duration":47.229362010999999,"stat_sum":{"num_bytes":3016618436,"num_objects":3825,"num_object_clones":3119,"num_object_copies":7650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3825,"num_whiteouts":0,"num_read":135806,"num_read_kb":504587,"num_write":127812,"num_write_kb":510763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"c"}]},{"pgid":"2.0","version":"112'132026","reported_seq":253676,"reported_epoch":112,"state":"active+clean","last_fresh":"2023-10-03T18:06:55.458270+0000","last_change":"2023-10-03T18:06:55.214657+0000","last_active":"2023-10-03T18:06:55.458270+0000","last_peered":"2023-10-03T18:06:55.458270+0000","last_clean":"2023-10-03T18:06:55.458270+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:06:55.458270+0000","last_undegraded":"2023-10-03T18:06:55.458270+0000","last_fullsized":"2023-10-03T18:06:55.458270+0000","mapping_epoch":12,"log_start":"97'121962","ondisk_log_start":"97'121962","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":545,"snaptrim_duration":34.962907455,"stat_sum":{"num_bytes":2915500788,"num_objects":3211,"num_object_clones":2524,"num_object_copies":6422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3211,"num_whiteouts":0,"num_read":127915,"num_read_kb":485645,"num_write":123916,"num_write_kb":495366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":266,"reported_epoch":111,"state":"active+clean","last_fresh":"2023-10-03T18:06:05.668003+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:06:05.668003+0000","last_peered":"2023-10-03T18:06:05.668003+0000","last_clean":"2023-10-03T18:06:05.668003+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:06:05.668003+0000","last_undegraded":"2023-10-03T18:06:05.668003+0000","last_fullsized":"2023-10-03T18:06:05.668003+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"112'134486","reported_seq":258678,"reported_epoch":112,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:06:57.225010+0000","last_change":"2023-10-03T18:06:55.761608+0000","last_active":"2023-10-03T18:06:57.225010+0000","last_peered":"2023-10-03T18:06:57.225010+0000","last_clean":"2023-10-03T18:06:57.225010+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:06:57.225010+0000","last_undegraded":"2023-10-03T18:06:57.225010+0000","last_fullsized":"2023-10-03T18:06:57.225010+0000","mapping_epoch":12,"log_start":"97'124465","ondisk_log_start":"97'124465","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":367,"snaptrim_duration":49.059865975000001,"stat_sum":{"num_bytes":2964415256,"num_objects":3744,"num_object_clones":3046,"num_object_copies":7488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3744,"num_whiteouts":0,"num_read":130817,"num_read_kb":495816,"num_write":125447,"num_write_kb":501454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"e"}]},{"pgid":"2.1","version":"112'121718","reported_seq":235221,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:55.452008+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:06:55.452008+0000","last_peered":"2023-10-03T18:06:55.452008+0000","last_clean":"2023-10-03T18:06:55.452008+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:06:55.452008+0000","last_undegraded":"2023-10-03T18:06:55.452008+0000","last_fullsized":"2023-10-03T18:06:55.452008+0000","mapping_epoch":12,"log_start":"93'111637","ondisk_log_start":"93'111637","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2094,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 316s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2701337199,"num_objects":4273,"num_object_clones":3638,"num_object_copies":8546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4273,"num_whiteouts":0,"num_read":119917,"num_read_kb":448789,"num_write":114375,"num_write_kb":457101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"112'122134","reported_seq":238767,"reported_epoch":112,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:06:57.159670+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:06:57.159670+0000","last_peered":"2023-10-03T18:06:57.159670+0000","last_clean":"2023-10-03T18:06:57.159670+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:06:57.159670+0000","last_undegraded":"2023-10-03T18:06:57.159670+0000","last_fullsized":"2023-10-03T18:06:57.159670+0000","mapping_epoch":12,"log_start":"93'112046","ondisk_log_start":"93'112046","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2346,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 324s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2704282473,"num_objects":4288,"num_object_clones":3644,"num_object_copies":8576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4288,"num_whiteouts":0,"num_read":124434,"num_read_kb":456218,"num_write":115324,"num_write_kb":460685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"112'122609","reported_seq":236248,"reported_epoch":112,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:06:57.250776+0000","last_change":"2023-10-03T18:06:56.590082+0000","last_active":"2023-10-03T18:06:57.250776+0000","last_peered":"2023-10-03T18:06:57.250776+0000","last_clean":"2023-10-03T18:06:57.250776+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:06:57.250776+0000","last_undegraded":"2023-10-03T18:06:57.250776+0000","last_fullsized":"2023-10-03T18:06:57.250776+0000","mapping_epoch":12,"log_start":"97'112565","ondisk_log_start":"97'112565","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":392,"snaptrim_duration":48.277408092000002,"stat_sum":{"num_bytes":2691908171,"num_objects":3322,"num_object_clones":2687,"num_object_copies":6644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":119592,"num_read_kb":449433,"num_write":113931,"num_write_kb":455372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"10"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22353700113,"num_objects":30242,"num_object_clones":24978,"num_object_copies":60484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30242,"num_whiteouts":0,"num_read":996174,"num_read_kb":3737689,"num_write":947797,"num_write_kb":3788065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7757467648,"data_stored":41763797538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38147,"internal_metadata":0},"log_size":80411,"ondisk_log_size":80411,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738779,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3272028,"kb_used_data":1454228,"kb_used_omap":1,"kb_used_meta":1817790,"kb_avail":91099812,"statfs":{"total":96636764160,"available":93286207488,"internally_reserved":0,"allocated":1489129472,"data_stored":7734017168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1861417414},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,7],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":2.036,"15min":2.343},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.8029999999999999,"5min":10.145,"15min":18.648},"last":5.9489999999999998},{"interface":"front","average":{"1min":1.8740000000000001,"5min":2.2130000000000001,"15min":2.8399999999999999},"min":{"1min":0.26200000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.0300000000000002,"5min":12.414999999999999,"15min":21.827999999999999},"last":5.8079999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":2.125,"15min":2.4209999999999998},"min":{"1min":0.30399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.936,"5min":13.945,"15min":26.074000000000002},"last":1.405},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.954,"15min":2.0619999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":19.097000000000001,"5min":19.097000000000001,"15min":19.097000000000001},"last":1.3400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738779,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3360496,"kb_used_data":2334248,"kb_used_omap":347,"kb_used_meta":1025892,"kb_avail":91011344,"statfs":{"total":96636764160,"available":93195616256,"internally_reserved":0,"allocated":2390269952,"data_stored":13163156171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1050513584},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,4,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:06:07 2023","interfaces":[{"interface":"back","average":{"1min":1.9419999999999999,"5min":3.8839999999999999,"15min":3.8540000000000001},"min":{"1min":0.17799999999999999,"5min":0.17299999999999999,"15min":0.16300000000000001},"max":{"1min":8.9849999999999994,"5min":30.852,"15min":30.852},"last":4.585},{"interface":"front","average":{"1min":1.5649999999999999,"5min":2.8300000000000001,"15min":3.4470000000000001},"min":{"1min":0.29399999999999998,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":5.4969999999999999,"5min":16.584,"15min":24.215},"last":4.4340000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:06:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":2.3079999999999998,"15min":2.2610000000000001},"min":{"1min":0.215,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.4779999999999998,"5min":14.195,"15min":14.195},"last":3.7490000000000001},{"interface":"front","average":{"1min":1.464,"5min":2.1160000000000001,"15min":2.3100000000000001},"min":{"1min":0.26600000000000001,"5min":0.214,"15min":0.20599999999999999},"max":{"1min":5.4889999999999999,"5min":10.949,"15min":13.677},"last":0.317}]}]},{"osd":1,"up_from":8,"seq":34359738771,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4957092,"kb_used_data":3789724,"kb_used_omap":1915,"kb_used_meta":1165444,"kb_avail":89414748,"statfs":{"total":96636764160,"available":91560701952,"internally_reserved":0,"allocated":3880677376,"data_stored":20885269318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1193415601},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,7,8,14,9,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.7919999999999998,"5min":2.7320000000000002,"15min":3.2909999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":9.9220000000000006,"5min":16.997,"15min":16.997},"last":0.40000000000000002},{"interface":"front","average":{"1min":2.5529999999999999,"5min":3.169,"15min":3.5529999999999999},"min":{"1min":0.216,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9779999999999998,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.29099999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":2.7320000000000002,"15min":2.6850000000000001},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":12.140000000000001,"5min":17.492999999999999,"15min":24.579000000000001},"last":12.140000000000001},{"interface":"front","average":{"1min":2.448,"5min":2.6579999999999999,"15min":2.907},"min":{"1min":0.252,"5min":0.215,"15min":0.17299999999999999},"max":{"1min":10.031000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":5.3739999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2389794816,"data_stored":13147737238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3879284736,"data_stored":20885275921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1488388096,"data_stored":7730784379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T18:06:59.208 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.208 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.208 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:06:59.209 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:06:59.837 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:59.837+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:06:59.892 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:59.890+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:06:59.893 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:06:59.893+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:07:00.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.008+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:00.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.032+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:00.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.243+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:00.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.243+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:00.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.369+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:00.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.369+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:00.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.369+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:00.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.369+0000 7fc515f98700 -1 reset not still connected to 0x562208acc820 2023-10-03T18:07:00.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.374+0000 7fc515f98700 -1 reset not still connected to 0x562208acde10 2023-10-03T18:07:00.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.447+0000 7fc515f98700 -1 reset not still connected to 0x562208acc820 2023-10-03T18:07:00.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.447+0000 7fc515f98700 -1 reset not still connected to 0x562208acde10 2023-10-03T18:07:00.575 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.575+0000 7fc515f98700 -1 reset not still connected to 0x562218489110 2023-10-03T18:07:00.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:00.578+0000 7fc515f98700 -1 reset not still connected to 0x562227189d40 2023-10-03T18:07:03.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:03.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.814+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:03.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.836+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:03.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.852+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:03.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.852+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:03.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.880+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:03.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.898+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:03.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:03.898+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:04.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.483+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:04.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.510+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:04.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.578+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:04.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.642+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:04.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.780+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:04.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.842+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:04.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.842+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:04.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.855+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:04.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.855+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:04.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.861+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:04.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.861+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:07:04.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.861+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:04.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:04.861+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:05.019 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:05.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.016+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:05.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.063+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:05.114 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.108+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:05.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.108+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:05.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.189+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:05.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.189+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:05.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.190+0000 7fc515f98700 -1 reset not still connected to 0x5622110bd860 2023-10-03T18:07:05.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.273+0000 7fc515f98700 -1 reset not still connected to 0x562215139790 2023-10-03T18:07:05.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:05.397+0000 7fc515f98700 -1 reset not still connected to 0x562219b7f520 2023-10-03T18:07:06.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.064+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:06.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.109+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:06.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.109+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:06.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.109+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:06.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.115+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:06.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.239+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:06.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.239+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:06.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.242+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:06.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.303+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:06.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:06.329+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:07.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.456+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:07.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.556+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:07.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.579+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:07.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.635+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:07.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.635+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:07.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.678+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:07.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.849+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:07.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:07.855+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:09.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.125+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:09.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.163+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:09.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.163+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:09.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.206+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:09.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.302+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:09.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.360+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:09.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.360+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:09.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.400+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:09.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.400+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:09.442 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:09.435+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:11.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:11.717+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:11.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:11.717+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:11.820 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:11.820+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:11.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:11.821+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:11.858 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:11.857+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:12.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:12.007+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:12.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:12.036+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:12.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:12.039+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:12.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:12.091+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:12.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:12.186+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:16.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.721+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:16.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.721+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:16.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.805+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:16.820 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.819+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:16.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.859+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:16.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.951+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:16.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.951+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:16.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:16.965+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:17.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:17.013+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:17.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:17.103+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:19.210 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:07:19.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:07:19.917 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:07:19.944 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1147,"stamp":"2023-10-03T18:07:18.622357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22373508913,"num_objects":31746,"num_object_clones":26480,"num_object_copies":63492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31746,"num_whiteouts":0,"num_read":1004058,"num_read_kb":3759160,"num_write":952813,"num_write_kb":3808755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80502,"ondisk_log_size":80502,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11894880,"kb_used_data":7617024,"kb_used_omap":2264,"kb_used_meta":4275559,"kb_avail":271220640,"statfs":{"total":289910292480,"available":277729935360,"internally_reserved":0,"allocated":7799832576,"data_stored":44145152155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2318809,"internal_metadata":4378172967},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,10,18,13,20,9,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10977382,"num_objects":1439,"num_object_clones":1437,"num_object_copies":2878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1439,"num_whiteouts":0,"num_read":3653,"num_read_kb":11571,"num_write":2763,"num_write_kb":10728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.744358"},"pg_stats":[{"pgid":"2.7","version":"119'127446","reported_seq":245118,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:14.396879+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:07:14.396879+0000","last_peered":"2023-10-03T18:07:14.396879+0000","last_clean":"2023-10-03T18:07:14.396879+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:07:14.396879+0000","last_undegraded":"2023-10-03T18:07:14.396879+0000","last_fullsized":"2023-10-03T18:07:14.396879+0000","mapping_epoch":12,"log_start":"97'117442","ondisk_log_start":"97'117442","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2119,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 337s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2788786611,"num_objects":4652,"num_object_clones":4002,"num_object_copies":9304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4652,"num_whiteouts":0,"num_read":123625,"num_read_kb":468384,"num_write":118679,"num_write_kb":474360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'116731","reported_seq":225739,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:14.337329+0000","last_change":"2023-10-03T18:07:11.822813+0000","last_active":"2023-10-03T18:07:14.337329+0000","last_peered":"2023-10-03T18:07:14.337329+0000","last_clean":"2023-10-03T18:07:14.337329+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:07:14.337329+0000","last_undegraded":"2023-10-03T18:07:14.337329+0000","last_fullsized":"2023-10-03T18:07:14.337329+0000","mapping_epoch":12,"log_start":"97'106667","ondisk_log_start":"97'106667","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":27,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":53,"snaptrim_duration":4.8904205879999996,"stat_sum":{"num_bytes":2575172156,"num_objects":3285,"num_object_clones":2682,"num_object_copies":6570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":115625,"num_read_kb":433394,"num_write":109406,"num_write_kb":437211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"3"}]},{"pgid":"2.5","version":"119'141098","reported_seq":268871,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:14.382090+0000","last_change":"2023-10-03T18:07:13.455501+0000","last_active":"2023-10-03T18:07:14.382090+0000","last_peered":"2023-10-03T18:07:14.382090+0000","last_clean":"2023-10-03T18:07:14.382090+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:07:14.382090+0000","last_undegraded":"2023-10-03T18:07:14.382090+0000","last_fullsized":"2023-10-03T18:07:14.382090+0000","mapping_epoch":12,"log_start":"97'131060","ondisk_log_start":"97'131060","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":49,"snaptrim_duration":6.5280616220000001,"stat_sum":{"num_bytes":3018916343,"num_objects":4003,"num_object_clones":3296,"num_object_copies":8006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4003,"num_whiteouts":0,"num_read":136981,"num_read_kb":507467,"num_write":128412,"num_write_kb":513044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"10"}]},{"pgid":"2.0","version":"119'133476","reported_seq":255537,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:15.492003+0000","last_change":"2023-10-03T18:07:14.440600+0000","last_active":"2023-10-03T18:07:15.492003+0000","last_peered":"2023-10-03T18:07:15.492003+0000","last_clean":"2023-10-03T18:07:15.492003+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:07:15.492003+0000","last_undegraded":"2023-10-03T18:07:15.492003+0000","last_fullsized":"2023-10-03T18:07:15.492003+0000","mapping_epoch":12,"log_start":"97'123470","ondisk_log_start":"97'123470","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":125,"snaptrim_duration":7.562371669,"stat_sum":{"num_bytes":2918400655,"num_objects":3301,"num_object_clones":2616,"num_object_copies":6602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3301,"num_whiteouts":0,"num_read":128903,"num_read_kb":488666,"num_write":124638,"num_write_kb":498181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":273,"reported_epoch":118,"state":"active+clean","last_fresh":"2023-10-03T18:07:05.981303+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:07:05.981303+0000","last_peered":"2023-10-03T18:07:05.981303+0000","last_clean":"2023-10-03T18:07:05.981303+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:07:05.981303+0000","last_undegraded":"2023-10-03T18:07:05.981303+0000","last_fullsized":"2023-10-03T18:07:05.981303+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'135617","reported_seq":260201,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:14.366227+0000","last_change":"2023-10-03T18:07:11.828311+0000","last_active":"2023-10-03T18:07:14.366227+0000","last_peered":"2023-10-03T18:07:14.366227+0000","last_clean":"2023-10-03T18:07:14.366227+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:07:14.366227+0000","last_undegraded":"2023-10-03T18:07:14.366227+0000","last_fullsized":"2023-10-03T18:07:14.366227+0000","mapping_epoch":12,"log_start":"97'125573","ondisk_log_start":"97'125573","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":31,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":55,"snaptrim_duration":2.2996420390000001,"stat_sum":{"num_bytes":2966828001,"num_objects":3889,"num_object_clones":3187,"num_object_copies":7778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3889,"num_whiteouts":0,"num_read":131729,"num_read_kb":498528,"num_write":126053,"num_write_kb":503794,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"11"}]},{"pgid":"2.1","version":"119'122821","reported_seq":236851,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:15.524721+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:07:15.524721+0000","last_peered":"2023-10-03T18:07:15.524721+0000","last_clean":"2023-10-03T18:07:15.524721+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:07:15.524721+0000","last_undegraded":"2023-10-03T18:07:15.524721+0000","last_fullsized":"2023-10-03T18:07:15.524721+0000","mapping_epoch":12,"log_start":"97'112741","ondisk_log_start":"97'112741","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2303,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 336s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2703868626,"num_objects":4544,"num_object_clones":3907,"num_object_copies":9088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4544,"num_whiteouts":0,"num_read":121000,"num_read_kb":451745,"num_write":115028,"num_write_kb":459599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"119'123237","reported_seq":240378,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:17.225591+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:07:17.225591+0000","last_peered":"2023-10-03T18:07:17.225591+0000","last_clean":"2023-10-03T18:07:17.225591+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:07:17.225591+0000","last_undegraded":"2023-10-03T18:07:17.225591+0000","last_fullsized":"2023-10-03T18:07:17.225591+0000","mapping_epoch":12,"log_start":"97'113150","ondisk_log_start":"97'113150","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2518,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 344s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2706793272,"num_objects":4597,"num_object_clones":3954,"num_object_copies":9194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":125648,"num_read_kb":459110,"num_write":115970,"num_write_kb":463128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"119'123650","reported_seq":237655,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:14.337979+0000","last_change":"2023-10-03T18:07:13.456488+0000","last_active":"2023-10-03T18:07:14.337979+0000","last_peered":"2023-10-03T18:07:14.337979+0000","last_clean":"2023-10-03T18:07:14.337979+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:07:14.337979+0000","last_undegraded":"2023-10-03T18:07:14.337979+0000","last_fullsized":"2023-10-03T18:07:14.337979+0000","mapping_epoch":12,"log_start":"97'113570","ondisk_log_start":"97'113570","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":2.3847828519999998,"stat_sum":{"num_bytes":2694152881,"num_objects":3473,"num_object_clones":2836,"num_object_copies":6946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3473,"num_whiteouts":0,"num_read":120441,"num_read_kb":451775,"num_write":114490,"num_write_kb":457544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22372918545,"num_objects":31744,"num_object_clones":26480,"num_object_copies":63488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31744,"num_whiteouts":0,"num_read":1003952,"num_read_kb":3759069,"num_write":952676,"num_write_kb":3806861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7796514816,"data_stored":44216003106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80403,"ondisk_log_size":80403,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738783,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3347280,"kb_used_data":1462472,"kb_used_omap":1,"kb_used_meta":1884798,"kb_avail":91024560,"statfs":{"total":96636764160,"available":93209149440,"internally_reserved":0,"allocated":1497571328,"data_stored":8140160853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":1930033606},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,0,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":2.036,"15min":2.343},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.8029999999999999,"5min":10.145,"15min":18.648},"last":3.0369999999999999},{"interface":"front","average":{"1min":1.8740000000000001,"5min":2.2130000000000001,"15min":2.8399999999999999},"min":{"1min":0.26200000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.0300000000000002,"5min":12.414999999999999,"15min":21.827999999999999},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":2.125,"15min":2.4209999999999998},"min":{"1min":0.30399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.936,"5min":13.945,"15min":26.074000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.954,"15min":2.0619999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":19.097000000000001,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738783,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3454520,"kb_used_data":2347760,"kb_used_omap":347,"kb_used_meta":1106404,"kb_avail":90917320,"statfs":{"total":96636764160,"available":93099335680,"internally_reserved":0,"allocated":2404106240,"data_stored":14036890626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1132957872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,12,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0219999999999998,"5min":3.4900000000000002,"15min":3.855},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16300000000000001},"max":{"1min":15.081,"5min":30.852,"15min":30.852},"last":5.875},{"interface":"front","average":{"1min":3.2050000000000001,"5min":2.8900000000000001,"15min":3.4710000000000001},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":12.289999999999999,"5min":16.584,"15min":24.215},"last":1.0429999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":2.1579999999999999,"15min":2.2170000000000001},"min":{"1min":0.30199999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.0720000000000001,"5min":14.195,"15min":14.195},"last":1.026},{"interface":"front","average":{"1min":2.129,"5min":2.0569999999999999,"15min":2.3380000000000001},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.042999999999999,"5min":12.042999999999999,"15min":13.677},"last":1.081}]}]},{"osd":1,"up_from":8,"seq":34359738774,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5093080,"kb_used_data":3806792,"kb_used_omap":1915,"kb_used_meta":1284356,"kb_avail":89278760,"statfs":{"total":96636764160,"available":91421450240,"internally_reserved":0,"allocated":3898155008,"data_stored":21968100676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1315181489},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,6,4,12,8,9,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.7919999999999998,"5min":2.7320000000000002,"15min":3.2909999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":9.9220000000000006,"5min":16.997,"15min":16.997},"last":0.255},{"interface":"front","average":{"1min":2.5529999999999999,"5min":3.169,"15min":3.5529999999999999},"min":{"1min":0.216,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9779999999999998,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.64700000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":2.7320000000000002,"15min":2.6850000000000001},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":12.140000000000001,"5min":17.492999999999999,"15min":24.579000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":2.448,"5min":2.6579999999999999,"15min":2.907},"min":{"1min":0.252,"5min":0.215,"15min":0.17299999999999999},"max":{"1min":10.031000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.28399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2402738176,"data_stored":14053874639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3897143296,"data_stored":22019495185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1496633344,"data_stored":8142633282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:07:19.946 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.946 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.946 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.946 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.946 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.947 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.947 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.947 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:19.947 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:07:21.171 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:21.166+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:07:21.206 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:21.205+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:07:21.206 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:21.205+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:07:22.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.026+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:22.098 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.097+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:22.098 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.097+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:22.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.161+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:22.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.173+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:22.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.300+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:22.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.445+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:22.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.452+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:22.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.452+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:22.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.481+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:22.625 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.624+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:22.625 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.624+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:22.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.650+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:22.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.739+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:22.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:22.776+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:25.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.137+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:07:25.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.153+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:25.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.153+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:25.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.193+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:25.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.193+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:25.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.274+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:25.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.274+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:25.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.274+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:25.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.274+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:25.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.366+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:25.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:25.366+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:32.121 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.118+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:32.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.142+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:32.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.142+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:32.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.143+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:32.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.220+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:32.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.220+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:32.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.222+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:32.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.222+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:32.279 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:32.277+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:35.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.679+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:35.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.681+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:35.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.731+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:35.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.731+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:35.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.735+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:35.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.881+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:35.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:35.881+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:37.306 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:37.304+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:07:37.316 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:37.312+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:07:37.316 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:07:37.312+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:07:39.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:39.397+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:39.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:07:40.665 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:07:40.670 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:07:40.689 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1158,"stamp":"2023-10-03T18:07:40.639061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22396471089,"num_objects":32847,"num_object_clones":27581,"num_object_copies":65694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32847,"num_whiteouts":0,"num_read":1009849,"num_read_kb":3782228,"num_write":958417,"num_write_kb":3831171,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80631,"ondisk_log_size":80631,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10846128,"kb_used_data":7661464,"kb_used_omap":3232,"kb_used_meta":3181407,"kb_avail":272269392,"statfs":{"total":289910292480,"available":278803857408,"internally_reserved":0,"allocated":7845339136,"data_stored":45855883419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":3257761743},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,4,9,11,19,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11771904,"num_objects":329,"num_object_clones":329,"num_object_copies":658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":329,"num_whiteouts":0,"num_read":2944,"num_read_kb":11716,"num_write":2878,"num_write_kb":11512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008164"},"pg_stats":[{"pgid":"2.7","version":"119'128609","reported_seq":246842,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:36.217001+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:07:36.217001+0000","last_peered":"2023-10-03T18:07:36.217001+0000","last_clean":"2023-10-03T18:07:36.217001+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:07:36.217001+0000","last_undegraded":"2023-10-03T18:07:36.217001+0000","last_fullsized":"2023-10-03T18:07:36.217001+0000","mapping_epoch":12,"log_start":"97'118542","ondisk_log_start":"97'118542","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2329,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 358s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2791645619,"num_objects":4920,"num_object_clones":4270,"num_object_copies":9840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":124430,"num_read_kb":471604,"num_write":119377,"num_write_kb":477152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'118074","reported_seq":227445,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:36.184055+0000","last_change":"2023-10-03T18:07:34.535766+0000","last_active":"2023-10-03T18:07:36.184055+0000","last_peered":"2023-10-03T18:07:36.184055+0000","last_clean":"2023-10-03T18:07:36.184055+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:07:36.184055+0000","last_undegraded":"2023-10-03T18:07:36.184055+0000","last_fullsized":"2023-10-03T18:07:36.184055+0000","mapping_epoch":12,"log_start":"97'107973","ondisk_log_start":"97'107973","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":221,"snaptrim_duration":25.623670873999998,"stat_sum":{"num_bytes":2578203196,"num_objects":3377,"num_object_clones":2774,"num_object_copies":6754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3377,"num_whiteouts":0,"num_read":116307,"num_read_kb":436110,"num_write":110147,"num_write_kb":440175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"8"}]},{"pgid":"2.5","version":"119'142430","reported_seq":270691,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:36.103504+0000","last_change":"2023-10-03T18:07:33.693973+0000","last_active":"2023-10-03T18:07:36.103504+0000","last_peered":"2023-10-03T18:07:36.103504+0000","last_clean":"2023-10-03T18:07:36.103504+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:07:36.103504+0000","last_undegraded":"2023-10-03T18:07:36.103504+0000","last_fullsized":"2023-10-03T18:07:36.103504+0000","mapping_epoch":12,"log_start":"100'132361","ondisk_log_start":"100'132361","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":28,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":185,"snaptrim_duration":26.766533246000002,"stat_sum":{"num_bytes":3022041591,"num_objects":4150,"num_object_clones":3443,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":137787,"num_read_kb":510687,"num_write":129165,"num_write_kb":516056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"2"}]},{"pgid":"2.0","version":"119'135107","reported_seq":257500,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:35.687224+0000","last_change":"2023-10-03T18:07:33.849168+0000","last_active":"2023-10-03T18:07:35.687224+0000","last_peered":"2023-10-03T18:07:35.687224+0000","last_clean":"2023-10-03T18:07:35.687224+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:07:35.687224+0000","last_undegraded":"2023-10-03T18:07:35.687224+0000","last_fullsized":"2023-10-03T18:07:35.687224+0000","mapping_epoch":12,"log_start":"105'125085","ondisk_log_start":"105'125085","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":445,"snaptrim_duration":26.970942465,"stat_sum":{"num_bytes":2921341583,"num_objects":3242,"num_object_clones":2557,"num_object_copies":6484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3242,"num_whiteouts":0,"num_read":129611,"num_read_kb":491478,"num_write":125362,"num_write_kb":501077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"2"}]},{"pgid":"1.0","version":"88'99","reported_seq":274,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-03T18:07:06.930950+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:07:06.930950+0000","last_peered":"2023-10-03T18:07:06.930950+0000","last_clean":"2023-10-03T18:07:06.930950+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:07:06.930950+0000","last_undegraded":"2023-10-03T18:07:06.930950+0000","last_fullsized":"2023-10-03T18:07:06.930950+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'136973","reported_seq":262021,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:36.089671+0000","last_change":"2023-10-03T18:07:33.698457+0000","last_active":"2023-10-03T18:07:36.089671+0000","last_peered":"2023-10-03T18:07:36.089671+0000","last_clean":"2023-10-03T18:07:36.089671+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:07:36.089671+0000","last_undegraded":"2023-10-03T18:07:36.089671+0000","last_fullsized":"2023-10-03T18:07:36.089671+0000","mapping_epoch":12,"log_start":"97'126878","ondisk_log_start":"97'126878","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":26,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":221,"snaptrim_duration":24.664462878999998,"stat_sum":{"num_bytes":2969842657,"num_objects":3998,"num_object_clones":3296,"num_object_copies":7996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3998,"num_whiteouts":0,"num_read":132515,"num_read_kb":501656,"num_write":126790,"num_write_kb":506742,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"4"}]},{"pgid":"2.1","version":"119'123856","reported_seq":238329,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:35.662962+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:07:35.662962+0000","last_peered":"2023-10-03T18:07:35.662962+0000","last_clean":"2023-10-03T18:07:35.662962+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:07:35.662962+0000","last_undegraded":"2023-10-03T18:07:35.662962+0000","last_fullsized":"2023-10-03T18:07:35.662962+0000","mapping_epoch":12,"log_start":"97'113842","ondisk_log_start":"97'113842","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2506,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 357s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2706375378,"num_objects":4783,"num_object_clones":4146,"num_object_copies":9566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4783,"num_whiteouts":0,"num_read":121658,"num_read_kb":454361,"num_write":115640,"num_write_kb":462047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"119'124335","reported_seq":241987,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:37.491680+0000","last_change":"2023-10-03T18:01:32.812308+0000","last_active":"2023-10-03T18:07:37.491680+0000","last_peered":"2023-10-03T18:07:37.491680+0000","last_clean":"2023-10-03T18:07:37.491680+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:07:37.491680+0000","last_undegraded":"2023-10-03T18:07:37.491680+0000","last_fullsized":"2023-10-03T18:07:37.491680+0000","mapping_epoch":12,"log_start":"97'114250","ondisk_log_start":"97'114250","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2735,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 364s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050279800000000005,"stat_sum":{"num_bytes":2709558072,"num_objects":4815,"num_object_clones":4172,"num_object_copies":9630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4815,"num_whiteouts":0,"num_read":126343,"num_read_kb":461874,"num_write":116645,"num_write_kb":465828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"119'124857","reported_seq":239234,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:36.095803+0000","last_change":"2023-10-03T18:07:34.529993+0000","last_active":"2023-10-03T18:07:36.095803+0000","last_peered":"2023-10-03T18:07:36.095803+0000","last_clean":"2023-10-03T18:07:36.095803+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:07:36.095803+0000","last_undegraded":"2023-10-03T18:07:36.095803+0000","last_fullsized":"2023-10-03T18:07:36.095803+0000","mapping_epoch":12,"log_start":"97'114778","ondisk_log_start":"97'114778","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":177,"snaptrim_duration":27.610871316000001,"stat_sum":{"num_bytes":2696872625,"num_objects":3560,"num_object_clones":2923,"num_object_copies":7120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3560,"num_whiteouts":0,"num_read":121092,"num_read_kb":454367,"num_write":115154,"num_write_kb":460200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"7"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22395880721,"num_objects":32845,"num_object_clones":27581,"num_object_copies":65690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32845,"num_whiteouts":0,"num_read":1009743,"num_read_kb":3782137,"num_write":958280,"num_write_kb":3829277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7841923072,"data_stored":45849037346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80532,"ondisk_log_size":80532,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738787,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2103880,"kb_used_data":1470528,"kb_used_omap":969,"kb_used_meta":632374,"kb_avail":92267960,"statfs":{"total":96636764160,"available":94482391040,"internally_reserved":0,"allocated":1505820672,"data_stored":8360562517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":647551854},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,3,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":2.036,"15min":2.343},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.8029999999999999,"5min":10.145,"15min":18.648},"last":4.0839999999999996},{"interface":"front","average":{"1min":1.8740000000000001,"5min":2.2130000000000001,"15min":2.8399999999999999},"min":{"1min":0.26200000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.0300000000000002,"5min":12.414999999999999,"15min":21.827999999999999},"last":0.221}]},{"osd":1,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":3.157,"5min":2.125,"15min":2.4209999999999998},"min":{"1min":0.30399999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":13.936,"5min":13.945,"15min":26.074000000000002},"last":0.88500000000000001},{"interface":"front","average":{"1min":2.3719999999999999,"5min":1.954,"15min":2.0619999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.188},"max":{"1min":19.097000000000001,"5min":19.097000000000001,"15min":19.097000000000001},"last":3.8130000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738787,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3521796,"kb_used_data":2360892,"kb_used_omap":347,"kb_used_meta":1160548,"kb_avail":90850044,"statfs":{"total":96636764160,"available":93030445056,"internally_reserved":0,"allocated":2417553408,"data_stored":14566949890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1188401328},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0219999999999998,"5min":3.4900000000000002,"15min":3.855},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16300000000000001},"max":{"1min":15.081,"5min":30.852,"15min":30.852},"last":0.98999999999999999},{"interface":"front","average":{"1min":3.2050000000000001,"5min":2.8900000000000001,"15min":3.4710000000000001},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":12.289999999999999,"5min":16.584,"15min":24.215},"last":5.173}]},{"osd":2,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":2.1579999999999999,"15min":2.2170000000000001},"min":{"1min":0.30199999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.0720000000000001,"5min":14.195,"15min":14.195},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.129,"5min":2.0569999999999999,"15min":2.3380000000000001},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.042999999999999,"5min":12.042999999999999,"15min":13.677},"last":5.0949999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738778,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5220452,"kb_used_data":3830044,"kb_used_omap":1915,"kb_used_meta":1388484,"kb_avail":89151388,"statfs":{"total":96636764160,"available":91291021312,"internally_reserved":0,"allocated":3921965056,"data_stored":22928371012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1421808561},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,1,5,7,18,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.7919999999999998,"5min":2.7320000000000002,"15min":3.2909999999999999},"min":{"1min":0.19700000000000001,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":9.9220000000000006,"5min":16.997,"15min":16.997},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.5529999999999999,"5min":3.169,"15min":3.5529999999999999},"min":{"1min":0.216,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9779999999999998,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.29299999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:06:51 2023","interfaces":[{"interface":"back","average":{"1min":2.956,"5min":2.7320000000000002,"15min":2.6850000000000001},"min":{"1min":0.222,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":12.140000000000001,"5min":17.492999999999999,"15min":24.579000000000001},"last":0.316},{"interface":"front","average":{"1min":2.448,"5min":2.6579999999999999,"15min":2.907},"min":{"1min":0.252,"5min":0.215,"15min":0.17299999999999999},"max":{"1min":10.031000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.26000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2416353280,"data_stored":14564232143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3920613376,"data_stored":22925055249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1504956416,"data_stored":8359749954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:07:40.690 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.691 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.691 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.691 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.691 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.691 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.692 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.692 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:07:40.692 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:07:42.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.576+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:42.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.582+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:42.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.657+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:42.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.809+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:42.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.813+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:42.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.861+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:42.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.861+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:42.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:42.895+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:43.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:43.030+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:43.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:43.030+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:51.415 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:07:51.414+0000 7f5801100700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T18:07:51.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.817+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:51.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.897+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:51.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.897+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:51.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.954+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:51.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.954+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:51.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.954+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:51.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.967+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:51.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.967+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:51.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.967+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:51.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:51.967+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:52.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.022+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:52.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.022+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:52.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.022+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:52.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.022+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:52.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.022+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:52.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.097+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:52.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:52.097+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:53.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:53.753+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:53.754 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:53.753+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:55.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.865+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:55.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.866+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:55.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.886+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:55.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.886+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:55.909 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.908+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:55.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.912+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:55.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.912+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:55.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:55.984+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:56.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.024+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:56.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.024+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:56.744 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.739+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:56.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.776+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:56.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.783+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:56.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.783+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:56.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.783+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:56.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.783+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:56.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.803+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:56.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:56.803+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:58.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.226+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:58.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.226+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:58.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.226+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:58.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.257+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:58.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.288+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:58.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.288+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:58.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.291+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:58.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.368+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:58.530 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:58.526+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:07:59.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.211+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:07:59.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.211+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:07:59.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.211+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:07:59.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.298+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:07:59.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.312+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:07:59.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:07:59.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:07:59.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:07:59.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:07:59.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:07:59.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:07:59.372+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:00.693 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:08:01.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:08:01.527 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:08:01.548 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1169,"stamp":"2023-10-03T18:08:00.748033+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22422738737,"num_objects":32515,"num_object_clones":27249,"num_object_copies":65030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32515,"num_whiteouts":0,"num_read":1017912,"num_read_kb":3809667,"num_write":964834,"num_write_kb":3856839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80489,"ondisk_log_size":80489,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11022464,"kb_used_data":7711396,"kb_used_omap":3232,"kb_used_meta":3307807,"kb_avail":272093056,"statfs":{"total":289910292480,"available":278623289344,"internally_reserved":0,"allocated":7896469504,"data_stored":45525962907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":3387195343},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,31,2,9,18,16],"upper_bound":256},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13668352,"num_objects":-291,"num_object_clones":-291,"num_object_copies":-582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-291,"num_whiteouts":0,"num_read":4794,"num_read_kb":14467,"num_write":3340,"num_write_kb":13360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.097305"},"pg_stats":[{"pgid":"2.7","version":"119'129641","reported_seq":248625,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:57.656831+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:07:57.656831+0000","last_peered":"2023-10-03T18:07:57.656831+0000","last_clean":"2023-10-03T18:07:57.656831+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:07:57.656831+0000","last_undegraded":"2023-10-03T18:07:57.656831+0000","last_fullsized":"2023-10-03T18:07:57.656831+0000","mapping_epoch":12,"log_start":"97'119543","ondisk_log_start":"97'119543","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2467,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 380s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2794893747,"num_objects":5023,"num_object_clones":4373,"num_object_copies":10046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5023,"num_whiteouts":0,"num_read":125375,"num_read_kb":474948,"num_write":120170,"num_write_kb":480324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"119'119265","reported_seq":229319,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:57.707314+0000","last_change":"2023-10-03T18:07:57.382802+0000","last_active":"2023-10-03T18:07:57.707314+0000","last_peered":"2023-10-03T18:07:57.707314+0000","last_clean":"2023-10-03T18:07:57.707314+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:07:57.707314+0000","last_undegraded":"2023-10-03T18:07:57.707314+0000","last_fullsized":"2023-10-03T18:07:57.707314+0000","mapping_epoch":12,"log_start":"100'109184","ondisk_log_start":"100'109184","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":405,"snaptrim_duration":50.450401649,"stat_sum":{"num_bytes":2581189180,"num_objects":3278,"num_object_clones":2675,"num_object_copies":6556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3278,"num_whiteouts":0,"num_read":117230,"num_read_kb":439327,"num_write":110878,"num_write_kb":443099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"5"}]},{"pgid":"2.5","version":"119'143802","reported_seq":272929,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:57.671095+0000","last_change":"2023-10-03T18:07:57.389205+0000","last_active":"2023-10-03T18:07:57.671095+0000","last_peered":"2023-10-03T18:07:57.671095+0000","last_clean":"2023-10-03T18:07:57.671095+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:07:57.671095+0000","last_undegraded":"2023-10-03T18:07:57.671095+0000","last_fullsized":"2023-10-03T18:07:57.671095+0000","mapping_epoch":12,"log_start":"105'133767","ondisk_log_start":"105'133767","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":357,"snaptrim_duration":48.084226229000002,"stat_sum":{"num_bytes":3025723895,"num_objects":4078,"num_object_clones":3371,"num_object_copies":8156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4078,"num_whiteouts":0,"num_read":139023,"num_read_kb":514751,"num_write":130065,"num_write_kb":519656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"7"}]},{"pgid":"2.0","version":"119'136305","reported_seq":259451,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:55.906396+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:07:55.906396+0000","last_peered":"2023-10-03T18:07:55.906396+0000","last_clean":"2023-10-03T18:07:55.906396+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:07:55.906396+0000","last_undegraded":"2023-10-03T18:07:55.906396+0000","last_fullsized":"2023-10-03T18:07:55.906396+0000","mapping_epoch":12,"log_start":"105'126288","ondisk_log_start":"105'126288","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":122,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":551,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2924597903,"num_objects":3220,"num_object_clones":2535,"num_object_copies":6440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3220,"num_whiteouts":0,"num_read":130578,"num_read_kb":494898,"num_write":126157,"num_write_kb":504257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":274,"reported_epoch":119,"state":"active+clean","last_fresh":"2023-10-03T18:07:06.930950+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:07:06.930950+0000","last_peered":"2023-10-03T18:07:06.930950+0000","last_clean":"2023-10-03T18:07:06.930950+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:07:06.930950+0000","last_undegraded":"2023-10-03T18:07:06.930950+0000","last_fullsized":"2023-10-03T18:07:06.930950+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"119'138325","reported_seq":264146,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:57.619080+0000","last_change":"2023-10-03T18:07:57.551458+0000","last_active":"2023-10-03T18:07:57.619080+0000","last_peered":"2023-10-03T18:07:57.619080+0000","last_clean":"2023-10-03T18:07:57.619080+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:07:57.619080+0000","last_undegraded":"2023-10-03T18:07:57.619080+0000","last_fullsized":"2023-10-03T18:07:57.619080+0000","mapping_epoch":12,"log_start":"105'128285","ondisk_log_start":"105'128285","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":401,"snaptrim_duration":48.726826758999998,"stat_sum":{"num_bytes":2973414369,"num_objects":3923,"num_object_clones":3221,"num_object_copies":7846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3923,"num_whiteouts":0,"num_read":133562,"num_read_kb":505376,"num_write":127663,"num_write_kb":510234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"}]},{"pgid":"2.1","version":"119'124851","reported_seq":240022,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:07:55.938356+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:07:55.938356+0000","last_peered":"2023-10-03T18:07:55.938356+0000","last_clean":"2023-10-03T18:07:55.938356+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:07:55.938356+0000","last_undegraded":"2023-10-03T18:07:55.938356+0000","last_fullsized":"2023-10-03T18:07:55.938356+0000","mapping_epoch":12,"log_start":"97'114842","ondisk_log_start":"97'114842","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2706,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 377s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2709275346,"num_objects":4881,"num_object_clones":4244,"num_object_copies":9762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":122605,"num_read_kb":457491,"num_write":116348,"num_write_kb":464879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"119'125745","reported_seq":244038,"reported_epoch":119,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:07:57.587428+0000","last_change":"2023-10-03T18:07:56.885797+0000","last_active":"2023-10-03T18:07:57.587428+0000","last_peered":"2023-10-03T18:07:57.587428+0000","last_clean":"2023-10-03T18:07:57.587428+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:07:57.587428+0000","last_undegraded":"2023-10-03T18:07:57.587428+0000","last_fullsized":"2023-10-03T18:07:57.587428+0000","mapping_epoch":12,"log_start":"97'115656","ondisk_log_start":"97'115656","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:55:29.716367+0000","scrub_duration":370.574046178,"objects_trimmed":252,"snaptrim_duration":49.974157925,"stat_sum":{"num_bytes":2712658744,"num_objects":4636,"num_object_clones":3993,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":127343,"num_read_kb":464941,"num_write":117402,"num_write_kb":468856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"5"}]},{"pgid":"2.4","version":"119'126207","reported_seq":241304,"reported_epoch":119,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:07:57.698871+0000","last_change":"2023-10-03T18:07:57.470919+0000","last_active":"2023-10-03T18:07:57.698871+0000","last_peered":"2023-10-03T18:07:57.698871+0000","last_clean":"2023-10-03T18:07:57.698871+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:07:57.698871+0000","last_undegraded":"2023-10-03T18:07:57.698871+0000","last_fullsized":"2023-10-03T18:07:57.698871+0000","mapping_epoch":12,"log_start":"105'116186","ondisk_log_start":"105'116186","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":365,"snaptrim_duration":50.551798681999998,"stat_sum":{"num_bytes":2700395185,"num_objects":3474,"num_object_clones":2837,"num_object_copies":6948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3474,"num_whiteouts":0,"num_read":122090,"num_read_kb":457844,"num_write":116014,"num_write_kb":463640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22422148369,"num_objects":32513,"num_object_clones":27249,"num_object_copies":65026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32513,"num_whiteouts":0,"num_read":1017806,"num_read_kb":3809576,"num_write":964697,"num_write_kb":3854945,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7893995520,"data_stored":45475752482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80390,"ondisk_log_size":80390,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738791,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2140912,"kb_used_data":1479784,"kb_used_omap":969,"kb_used_meta":660150,"kb_avail":92230928,"statfs":{"total":96636764160,"available":94444470272,"internally_reserved":0,"allocated":1515298816,"data_stored":8375328597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":675994478},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.7080000000000002,"5min":2.387,"15min":2.3540000000000001},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":8.3610000000000007,"5min":10.145,"15min":18.648},"last":2.8799999999999999},{"interface":"front","average":{"1min":2.891,"5min":2.1499999999999999,"15min":2.8969999999999998},"min":{"1min":0.221,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":16.285,"5min":16.285,"15min":21.827999999999999},"last":2.4580000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.6200000000000001,"5min":2.1600000000000001,"15min":2.4849999999999999},"min":{"1min":0.30599999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.499000000000001,"5min":13.945,"15min":26.074000000000002},"last":12.460000000000001},{"interface":"front","average":{"1min":2.0939999999999999,"5min":1.8959999999999999,"15min":2.089},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":12,"5min":19.097000000000001,"15min":19.097000000000001},"last":12.436999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738791,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3564724,"kb_used_data":2376044,"kb_used_omap":347,"kb_used_meta":1188324,"kb_avail":90807116,"statfs":{"total":96636764160,"available":92986486784,"internally_reserved":0,"allocated":2433069056,"data_stored":14385484802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1216843952},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0219999999999998,"5min":3.4900000000000002,"15min":3.855},"min":{"1min":0.17899999999999999,"5min":0.17799999999999999,"15min":0.16300000000000001},"max":{"1min":15.081,"5min":30.852,"15min":30.852},"last":7.0110000000000001},{"interface":"front","average":{"1min":3.2050000000000001,"5min":2.8900000000000001,"15min":3.4710000000000001},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.19400000000000001},"max":{"1min":12.289999999999999,"5min":16.584,"15min":24.215},"last":2.8929999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:07:10 2023","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":2.1579999999999999,"15min":2.2170000000000001},"min":{"1min":0.30199999999999999,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":4.0720000000000001,"5min":14.195,"15min":14.195},"last":0.33100000000000002},{"interface":"front","average":{"1min":2.129,"5min":2.0569999999999999,"15min":2.3380000000000001},"min":{"1min":0.27500000000000002,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.042999999999999,"5min":12.042999999999999,"15min":13.677},"last":0.25700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738782,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5316828,"kb_used_data":3855568,"kb_used_omap":1915,"kb_used_meta":1459332,"kb_avail":89055012,"statfs":{"total":96636764160,"available":91192332288,"internally_reserved":0,"allocated":3948101632,"data_stored":22765149508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1494356913},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,4,4,29,0,7,13,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":2.5720000000000001,"15min":3.1619999999999999},"min":{"1min":0.255,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.467000000000001,"5min":16.997,"15min":16.997},"last":0.17599999999999999},{"interface":"front","average":{"1min":2.7480000000000002,"5min":3.097,"15min":3.5209999999999999},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9510000000000005,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.23899999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.5190000000000001,"15min":2.7120000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":3.726,"5min":17.492999999999999,"15min":24.579000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.133,"5min":2.649,"15min":2.8340000000000001},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.890000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.28000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2432094208,"data_stored":14360406991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3947528192,"data_stored":22738863377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1514373120,"data_stored":8376482114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:08:01.549 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.549 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.550 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.550 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.550 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.550 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.550 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:01.551 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:08:01.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.670+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:01.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.670+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:01.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.756+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:01.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.826+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:01.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.826+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:01.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.826+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:01.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.902+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:01.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.941+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:01.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:01.960+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:09.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.787+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:09.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.798+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:09.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.847+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:09.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.880+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:09.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.888+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:09.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.892+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:09.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.892+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:09.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.901+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:09.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.927+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:09.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:09.930+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:11.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.486+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:11.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.486+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:11.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.629+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:11.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.629+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:11.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.681+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:11.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:11.731+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:13.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.211+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:13.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.301+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:13.316 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.316+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:13.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.331+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:13.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.529+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:13.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.559+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:13.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.691+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:13.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.732+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:13.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:13.912+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:14.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.644+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:14.693 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.681+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:14.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.715+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:14.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.715+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:14.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.781+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:14.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.781+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:14.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.818+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:14.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.818+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:14.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:14.824+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:20.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.134+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:20.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.138+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:20.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.187+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:20.215 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.214+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:20.215 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.214+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:20.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.292+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:20.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.292+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:20.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.317+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:20.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.321+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:08:20.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.337+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:20.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.386+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:20.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.386+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:20.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.390+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:20.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.393+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:20.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.498+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:20.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.498+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:20.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.513+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:20.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.513+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:20.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:20.612+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:21.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:08:22.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:22.005+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:22.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:22.060+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:22.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:22.104+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:22.258 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:22.258+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:22.268 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:08:22.295 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:08:22.314 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1185,"stamp":"2023-10-03T18:08:20.667405+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22444750641,"num_objects":34348,"num_object_clones":29082,"num_object_copies":68696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34348,"num_whiteouts":0,"num_read":1026337,"num_read_kb":3833436,"num_write":970432,"num_write_kb":3878511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80437,"ondisk_log_size":80437,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11354308,"kb_used_data":7754540,"kb_used_omap":3232,"kb_used_meta":3596511,"kb_avail":271761212,"statfs":{"total":289910292480,"available":278283481088,"internally_reserved":0,"allocated":7940648960,"data_stored":48304320141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":3682828239},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,0,14,10,8,10,5,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13782938,"num_objects":1705,"num_object_clones":1707,"num_object_copies":3410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1705,"num_whiteouts":0,"num_read":3871,"num_read_kb":13593,"num_write":3423,"num_write_kb":13508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002082"},"pg_stats":[{"pgid":"2.7","version":"126'130890","reported_seq":250301,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:19.886373+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:08:19.886373+0000","last_peered":"2023-10-03T18:08:19.886373+0000","last_clean":"2023-10-03T18:08:19.886373+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:08:19.886373+0000","last_undegraded":"2023-10-03T18:08:19.886373+0000","last_fullsized":"2023-10-03T18:08:19.886373+0000","mapping_epoch":12,"log_start":"105'120847","ondisk_log_start":"105'120847","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2631,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 402s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2797760997,"num_objects":5396,"num_object_clones":4745,"num_object_copies":10792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5396,"num_whiteouts":0,"num_read":126359,"num_read_kb":477959,"num_write":120891,"num_write_kb":483140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'120516","reported_seq":230996,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:19.882560+0000","last_change":"2023-10-03T18:08:18.607674+0000","last_active":"2023-10-03T18:08:19.882560+0000","last_peered":"2023-10-03T18:08:19.882560+0000","last_clean":"2023-10-03T18:08:19.882560+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:08:19.882560+0000","last_undegraded":"2023-10-03T18:08:19.882560+0000","last_fullsized":"2023-10-03T18:08:19.882560+0000","mapping_epoch":12,"log_start":"105'110492","ondisk_log_start":"105'110492","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":77,"snaptrim_duration":11.715841656,"stat_sum":{"num_bytes":2583843438,"num_objects":3462,"num_object_clones":2858,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":118143,"num_read_kb":442144,"num_write":111539,"num_write_kb":445704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"}]},{"pgid":"2.5","version":"126'145274","reported_seq":274870,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:19.813630+0000","last_change":"2023-10-03T18:08:17.887147+0000","last_active":"2023-10-03T18:08:19.813630+0000","last_peered":"2023-10-03T18:08:19.813630+0000","last_clean":"2023-10-03T18:08:19.813630+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:08:19.813630+0000","last_undegraded":"2023-10-03T18:08:19.813630+0000","last_fullsized":"2023-10-03T18:08:19.813630+0000","mapping_epoch":12,"log_start":"105'135272","ondisk_log_start":"105'135272","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":66,"snaptrim_duration":8.0790073339999999,"stat_sum":{"num_bytes":3028824666,"num_objects":4327,"num_object_clones":3618,"num_object_copies":8654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4327,"num_whiteouts":0,"num_read":140334,"num_read_kb":518140,"num_write":130862,"num_write_kb":522715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"1"}]},{"pgid":"2.0","version":"126'137388","reported_seq":261000,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:16.168297+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:08:16.168297+0000","last_peered":"2023-10-03T18:08:16.168297+0000","last_clean":"2023-10-03T18:08:16.168297+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:08:16.168297+0000","last_undegraded":"2023-10-03T18:08:16.168297+0000","last_fullsized":"2023-10-03T18:08:16.168297+0000","mapping_epoch":12,"log_start":"105'127289","ondisk_log_start":"105'127289","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":257,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 31s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2927301365,"num_objects":3515,"num_object_clones":2828,"num_object_copies":7030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3515,"num_whiteouts":0,"num_read":131525,"num_read_kb":497745,"num_write":126839,"num_write_kb":506914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":281,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-03T18:08:05.875075+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:08:05.875075+0000","last_peered":"2023-10-03T18:08:05.875075+0000","last_clean":"2023-10-03T18:08:05.875075+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:08:05.875075+0000","last_undegraded":"2023-10-03T18:08:05.875075+0000","last_fullsized":"2023-10-03T18:08:05.875075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'139739","reported_seq":265915,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:19.771585+0000","last_change":"2023-10-03T18:08:17.887077+0000","last_active":"2023-10-03T18:08:19.771585+0000","last_peered":"2023-10-03T18:08:19.771585+0000","last_clean":"2023-10-03T18:08:19.771585+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:08:19.771585+0000","last_undegraded":"2023-10-03T18:08:19.771585+0000","last_fullsized":"2023-10-03T18:08:19.771585+0000","mapping_epoch":12,"log_start":"105'129689","ondisk_log_start":"105'129689","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":57,"snaptrim_duration":11.04859798,"stat_sum":{"num_bytes":2976326577,"num_objects":4150,"num_object_clones":3449,"num_object_copies":8300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4150,"num_whiteouts":0,"num_read":134535,"num_read_kb":508308,"num_write":128406,"num_write_kb":513109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"5"}]},{"pgid":"2.1","version":"126'125953","reported_seq":241620,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:16.178332+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:08:16.178332+0000","last_peered":"2023-10-03T18:08:16.178332+0000","last_clean":"2023-10-03T18:08:16.178332+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:08:16.178332+0000","last_undegraded":"2023-10-03T18:08:16.178332+0000","last_fullsized":"2023-10-03T18:08:16.178332+0000","mapping_epoch":12,"log_start":"104'115943","ondisk_log_start":"104'115943","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2902,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 397s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2711806423,"num_objects":5163,"num_object_clones":4531,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5163,"num_whiteouts":0,"num_read":123667,"num_read_kb":460337,"num_write":116998,"num_write_kb":467372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"126'127333","reported_seq":246006,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:17.772081+0000","last_change":"2023-10-03T18:08:17.713617+0000","last_active":"2023-10-03T18:08:17.772081+0000","last_peered":"2023-10-03T18:08:17.772081+0000","last_clean":"2023-10-03T18:08:17.772081+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:08:17.772081+0000","last_undegraded":"2023-10-03T18:08:17.772081+0000","last_fullsized":"2023-10-03T18:08:17.772081+0000","mapping_epoch":12,"log_start":"105'117269","ondisk_log_start":"105'117269","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:31.732949+0000","scrub_duration":370.574046178,"objects_trimmed":171,"snaptrim_duration":10.872068942,"stat_sum":{"num_bytes":2715153157,"num_objects":4631,"num_object_clones":3989,"num_object_copies":9262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4631,"num_whiteouts":0,"num_read":128557,"num_read_kb":467828,"num_write":118052,"num_write_kb":471322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"4"}]},{"pgid":"2.4","version":"126'127537","reported_seq":243053,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:19.823898+0000","last_change":"2023-10-03T18:08:18.722569+0000","last_active":"2023-10-03T18:08:19.823898+0000","last_peered":"2023-10-03T18:08:19.823898+0000","last_clean":"2023-10-03T18:08:19.823898+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:08:19.823898+0000","last_undegraded":"2023-10-03T18:08:19.823898+0000","last_fullsized":"2023-10-03T18:08:19.823898+0000","mapping_epoch":12,"log_start":"105'117491","ondisk_log_start":"105'117491","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":33,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":72,"snaptrim_duration":10.083376474,"stat_sum":{"num_bytes":2703143650,"num_objects":3702,"num_object_clones":3064,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3702,"num_whiteouts":0,"num_read":123111,"num_read_kb":460884,"num_write":116708,"num_write_kb":466341,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"7"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22444160273,"num_objects":34346,"num_object_clones":29082,"num_object_copies":68692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34346,"num_whiteouts":0,"num_read":1026231,"num_read_kb":3833345,"num_write":970295,"num_write_kb":3876617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7937519616,"data_stored":48389777954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80338,"ondisk_log_size":80338,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738795,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2193180,"kb_used_data":1487636,"kb_used_omap":969,"kb_used_meta":704566,"kb_avail":92178660,"statfs":{"total":96636764160,"available":94390947840,"internally_reserved":0,"allocated":1523339264,"data_stored":8960025398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":721476462},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.7080000000000002,"5min":2.387,"15min":2.3540000000000001},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":8.3610000000000007,"5min":10.145,"15min":18.648},"last":4.2709999999999999},{"interface":"front","average":{"1min":2.891,"5min":2.1499999999999999,"15min":2.8969999999999998},"min":{"1min":0.221,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":16.285,"5min":16.285,"15min":21.827999999999999},"last":3.698}]},{"osd":1,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.6200000000000001,"5min":2.1600000000000001,"15min":2.4849999999999999},"min":{"1min":0.30599999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.499000000000001,"5min":13.945,"15min":26.074000000000002},"last":4.085},{"interface":"front","average":{"1min":2.0939999999999999,"5min":1.8959999999999999,"15min":2.089},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":12,"5min":19.097000000000001,"15min":19.097000000000001},"last":3.891}]}]},{"osd":0,"up_from":8,"seq":34359738795,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3681524,"kb_used_data":2388780,"kb_used_omap":347,"kb_used_meta":1292388,"kb_avail":90690316,"statfs":{"total":96636764160,"available":92866883584,"internally_reserved":0,"allocated":2446110720,"data_stored":15089260909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1323405488},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":3.3839999999999999,"15min":3.798},"min":{"1min":0.215,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":9.3360000000000003,"5min":30.852,"15min":30.852},"last":0.191},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.8090000000000002,"15min":3.3450000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":12.077999999999999,"5min":16.584,"15min":24.215},"last":0.27800000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.607,"15min":2.145},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":8.2609999999999992,"5min":11.818,"15min":14.195},"last":0.29899999999999999},{"interface":"front","average":{"1min":2.028,"5min":2.085,"15min":2.2799999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.031000000000001,"5min":12.042999999999999,"15min":12.630000000000001},"last":0.88500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738786,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5479604,"kb_used_data":3878124,"kb_used_omap":1915,"kb_used_meta":1599556,"kb_avail":88892236,"statfs":{"total":96636764160,"available":91025649664,"internally_reserved":0,"allocated":3971198976,"data_stored":24255033834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1637946289},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,12,8,3,8,5,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":2.5720000000000001,"15min":3.1619999999999999},"min":{"1min":0.255,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.467000000000001,"5min":16.997,"15min":16.997},"last":2.871},{"interface":"front","average":{"1min":2.7480000000000002,"5min":3.097,"15min":3.5209999999999999},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9510000000000005,"5min":20.033000000000001,"15min":24.481000000000002},"last":5.633}]},{"osd":2,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.5190000000000001,"15min":2.7120000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":3.726,"5min":17.492999999999999,"15min":24.579000000000001},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.133,"5min":2.649,"15min":2.8340000000000001},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.890000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.33500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2444939264,"data_stored":15119756436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3970203648,"data_stored":24300479761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1522376704,"data_stored":8969541757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:08:22.316 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.316 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.317 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.317 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.317 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.318 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.318 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:22.318 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:08:23.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:23.110+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:23.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:23.110+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:23.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:23.223+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:23.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:23.237+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:23.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:23.252+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:23.457 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:08:23.457+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:08:23.470 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:08:23.469+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:08:29.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.812+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:29.814 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.812+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:29.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.816+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:29.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.852+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:29.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.852+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:29.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.865+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:29.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.869+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:29.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:29.898+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:30.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:30.955+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:08:31.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.163+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:31.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.164+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:31.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.189+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:31.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.339+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:31.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.343+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:31.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.347+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:31.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.347+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:31.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.375+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:31.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.430+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:31.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:31.430+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:33.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:33.940+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:34.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.040+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:34.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.077+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:34.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.077+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:34.093 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.092+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:34.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.102+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:34.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.102+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:34.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.197+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:34.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.197+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:34.275 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.274+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:34.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.753+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:34.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.760+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:34.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.760+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:34.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.782+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:34.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.796+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:34.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.796+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:34.868 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.866+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:34.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:34.866+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:35.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:35.112+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:36.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:36.787+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:36.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:36.863+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:38.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.020+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:38.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.076+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:38.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.128+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:38.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.492+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:38.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.500+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.526+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.526+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:08:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.526+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.526+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:38.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.529+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:38.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.529+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:38.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.565+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:38.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.758+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:38.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.758+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:38.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.811+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:38.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.811+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:38.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:38.839+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:41.629 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:08:41.628+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:08:41.630 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:08:41.628+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:08:41.630 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:08:41.628+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:08:42.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:08:42.997 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:08:43.004 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:08:43.026 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1196,"stamp":"2023-10-03T18:08:42.678687+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22469998385,"num_objects":35079,"num_object_clones":29813,"num_object_copies":70158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35079,"num_whiteouts":0,"num_read":1032574,"num_read_kb":3858240,"num_write":976575,"num_write_kb":3903083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80579,"ondisk_log_size":80579,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11580536,"kb_used_data":7802400,"kb_used_omap":3232,"kb_used_meta":3774879,"kb_avail":271534984,"statfs":{"total":289910292480,"available":278051823616,"internally_reserved":0,"allocated":7989657600,"data_stored":49630744205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":3865477071},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,3,6,14,10,18,6,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17797120,"num_objects":289,"num_object_clones":289,"num_object_copies":578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":0,"num_read":4419,"num_read_kb":17572,"num_write":4323,"num_write_kb":17292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005632"},"pg_stats":[{"pgid":"2.7","version":"126'132015","reported_seq":252047,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:41.963951+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:08:41.963951+0000","last_peered":"2023-10-03T18:08:41.963951+0000","last_clean":"2023-10-03T18:08:41.963951+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:08:41.963951+0000","last_undegraded":"2023-10-03T18:08:41.963951+0000","last_fullsized":"2023-10-03T18:08:41.963951+0000","mapping_epoch":12,"log_start":"105'121948","ondisk_log_start":"105'121948","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2810,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 424s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2800898533,"num_objects":5576,"num_object_clones":4925,"num_object_copies":11152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":127138,"num_read_kb":481059,"num_write":121657,"num_write_kb":486204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'121676","reported_seq":232615,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:41.872284+0000","last_change":"2023-10-03T18:08:41.663274+0000","last_active":"2023-10-03T18:08:41.872284+0000","last_peered":"2023-10-03T18:08:41.872284+0000","last_clean":"2023-10-03T18:08:41.872284+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:08:41.872284+0000","last_undegraded":"2023-10-03T18:08:41.872284+0000","last_fullsized":"2023-10-03T18:08:41.872284+0000","mapping_epoch":12,"log_start":"105'111596","ondisk_log_start":"105'111596","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":25,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":230,"snaptrim_duration":33.110557073999999,"stat_sum":{"num_bytes":2586575470,"num_objects":3485,"num_object_clones":2881,"num_object_copies":6970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3485,"num_whiteouts":0,"num_read":118828,"num_read_kb":444872,"num_write":112206,"num_write_kb":448372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"1"}]},{"pgid":"2.5","version":"126'146653","reported_seq":276875,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:41.902187+0000","last_change":"2023-10-03T18:08:41.295233+0000","last_active":"2023-10-03T18:08:41.902187+0000","last_peered":"2023-10-03T18:08:41.902187+0000","last_clean":"2023-10-03T18:08:41.902187+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:08:41.902187+0000","last_undegraded":"2023-10-03T18:08:41.902187+0000","last_fullsized":"2023-10-03T18:08:41.902187+0000","mapping_epoch":12,"log_start":"110'136580","ondisk_log_start":"110'136580","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":237,"snaptrim_duration":32.799070405000002,"stat_sum":{"num_bytes":3032031834,"num_objects":4364,"num_object_clones":3655,"num_object_copies":8728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4364,"num_whiteouts":0,"num_read":141209,"num_read_kb":521612,"num_write":131647,"num_write_kb":525855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"6"}]},{"pgid":"2.0","version":"126'138692","reported_seq":262937,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:41.256643+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:08:41.256643+0000","last_peered":"2023-10-03T18:08:41.256643+0000","last_clean":"2023-10-03T18:08:41.256643+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:08:41.256643+0000","last_undegraded":"2023-10-03T18:08:41.256643+0000","last_fullsized":"2023-10-03T18:08:41.256643+0000","mapping_epoch":12,"log_start":"112'128593","ondisk_log_start":"112'128593","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":480,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 56s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2930709237,"num_objects":3784,"num_object_clones":3097,"num_object_copies":7568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3784,"num_whiteouts":0,"num_read":132403,"num_read_kb":501241,"num_write":127671,"num_write_kb":510242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":281,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-03T18:08:05.875075+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:08:05.875075+0000","last_peered":"2023-10-03T18:08:05.875075+0000","last_clean":"2023-10-03T18:08:05.875075+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:08:05.875075+0000","last_undegraded":"2023-10-03T18:08:05.875075+0000","last_fullsized":"2023-10-03T18:08:05.875075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'141093","reported_seq":267839,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:41.755873+0000","last_change":"2023-10-03T18:08:41.663153+0000","last_active":"2023-10-03T18:08:41.755873+0000","last_peered":"2023-10-03T18:08:41.755873+0000","last_clean":"2023-10-03T18:08:41.755873+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:08:41.755873+0000","last_undegraded":"2023-10-03T18:08:41.755873+0000","last_fullsized":"2023-10-03T18:08:41.755873+0000","mapping_epoch":12,"log_start":"109'131092","ondisk_log_start":"109'131092","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":219,"snaptrim_duration":34.824675826000004,"stat_sum":{"num_bytes":2979652529,"num_objects":4196,"num_object_clones":3495,"num_object_copies":8392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4196,"num_whiteouts":0,"num_read":135362,"num_read_kb":511596,"num_write":129206,"num_write_kb":516309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"}]},{"pgid":"2.1","version":"126'127328","reported_seq":243577,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:41.274050+0000","last_change":"2023-10-03T18:01:38.620897+0000","last_active":"2023-10-03T18:08:41.274050+0000","last_peered":"2023-10-03T18:08:41.274050+0000","last_clean":"2023-10-03T18:08:41.274050+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:08:41.274050+0000","last_undegraded":"2023-10-03T18:08:41.274050+0000","last_fullsized":"2023-10-03T18:08:41.274050+0000","mapping_epoch":12,"log_start":"105'117246","ondisk_log_start":"105'117246","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3188,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 422s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040915,"stat_sum":{"num_bytes":2715320791,"num_objects":5408,"num_object_clones":4776,"num_object_copies":10816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5408,"num_whiteouts":0,"num_read":124467,"num_read_kb":463513,"num_write":117849,"num_write_kb":470776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"126'128902","reported_seq":247920,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:37.855031+0000","last_change":"2023-10-03T18:08:37.409540+0000","last_active":"2023-10-03T18:08:37.855031+0000","last_peered":"2023-10-03T18:08:37.855031+0000","last_clean":"2023-10-03T18:08:37.855031+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:08:37.855031+0000","last_undegraded":"2023-10-03T18:08:37.855031+0000","last_fullsized":"2023-10-03T18:08:37.855031+0000","mapping_epoch":12,"log_start":"105'118875","ondisk_log_start":"105'118875","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:31.732949+0000","scrub_duration":370.574046178,"objects_trimmed":486,"snaptrim_duration":30.567874569000001,"stat_sum":{"num_bytes":2718089989,"num_objects":4538,"num_object_clones":3896,"num_object_copies":9076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4538,"num_whiteouts":0,"num_read":129223,"num_read_kb":470480,"num_write":118763,"num_write_kb":474166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"e"}]},{"pgid":"2.4","version":"126'128746","reported_seq":244784,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:41.763599+0000","last_change":"2023-10-03T18:08:41.215403+0000","last_active":"2023-10-03T18:08:41.763599+0000","last_peered":"2023-10-03T18:08:41.763599+0000","last_clean":"2023-10-03T18:08:41.763599+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:08:41.763599+0000","last_undegraded":"2023-10-03T18:08:41.763599+0000","last_fullsized":"2023-10-03T18:08:41.763599+0000","mapping_epoch":12,"log_start":"105'118695","ondisk_log_start":"105'118695","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":27,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":216,"snaptrim_duration":34.364728067999998,"stat_sum":{"num_bytes":2706129634,"num_objects":3726,"num_object_clones":3088,"num_object_copies":7452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3726,"num_whiteouts":0,"num_read":123838,"num_read_kb":463776,"num_write":117439,"num_write_kb":469265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22469408017,"num_objects":35077,"num_object_clones":29813,"num_object_copies":70154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35077,"num_whiteouts":0,"num_read":1032468,"num_read_kb":3858149,"num_write":976438,"num_write_kb":3901189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7986667520,"data_stored":49630390818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80480,"ondisk_log_size":80480,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738800,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2196392,"kb_used_data":1497952,"kb_used_omap":969,"kb_used_meta":697462,"kb_avail":92175448,"statfs":{"total":96636764160,"available":94387658752,"internally_reserved":0,"allocated":1533902848,"data_stored":9459680054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":714201966},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,2,2,1,6,3,7],"upper_bound":1024},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.7080000000000002,"5min":2.387,"15min":2.3540000000000001},"min":{"1min":0.17499999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":8.3610000000000007,"5min":10.145,"15min":18.648},"last":3.1659999999999999},{"interface":"front","average":{"1min":2.891,"5min":2.1499999999999999,"15min":2.8969999999999998},"min":{"1min":0.221,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":16.285,"5min":16.285,"15min":21.827999999999999},"last":7.6719999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:07:52 2023","interfaces":[{"interface":"back","average":{"1min":2.6200000000000001,"5min":2.1600000000000001,"15min":2.4849999999999999},"min":{"1min":0.30599999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.499000000000001,"5min":13.945,"15min":26.074000000000002},"last":7.6310000000000002},{"interface":"front","average":{"1min":2.0939999999999999,"5min":1.8959999999999999,"15min":2.089},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":12,"5min":19.097000000000001,"15min":19.097000000000001},"last":7.4770000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738799,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3750260,"kb_used_data":2402860,"kb_used_omap":347,"kb_used_meta":1347044,"kb_avail":90621580,"statfs":{"total":96636764160,"available":92796497920,"internally_reserved":0,"allocated":2460528640,"data_stored":15382628717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1379373232},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":3.3839999999999999,"15min":3.798},"min":{"1min":0.215,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":9.3360000000000003,"5min":30.852,"15min":30.852},"last":3.0760000000000001},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.8090000000000002,"15min":3.3450000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":12.077999999999999,"5min":16.584,"15min":24.215},"last":4.8399999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.607,"15min":2.145},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":8.2609999999999992,"5min":11.818,"15min":14.195},"last":2.7759999999999998},{"interface":"front","average":{"1min":2.028,"5min":2.085,"15min":2.2799999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.031000000000001,"5min":12.042999999999999,"15min":12.630000000000001},"last":11.393000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738790,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5633884,"kb_used_data":3901588,"kb_used_omap":1915,"kb_used_meta":1730372,"kb_avail":88737956,"statfs":{"total":96636764160,"available":90867666944,"internally_reserved":0,"allocated":3995226112,"data_stored":24788435434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1771901873},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,3,8,8,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":2.5720000000000001,"15min":3.1619999999999999},"min":{"1min":0.255,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.467000000000001,"5min":16.997,"15min":16.997},"last":2.9660000000000002},{"interface":"front","average":{"1min":2.7480000000000002,"5min":3.097,"15min":3.5209999999999999},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9510000000000005,"5min":20.033000000000001,"15min":24.481000000000002},"last":3.6280000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.5190000000000001,"15min":2.7120000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":3.726,"5min":17.492999999999999,"15min":24.579000000000001},"last":3.71},{"interface":"front","average":{"1min":2.133,"5min":2.649,"15min":2.8340000000000001},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.890000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":0.26900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2458877952,"data_stored":15382600852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3994816512,"data_stored":24787559697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1532973056,"data_stored":9460230269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:08:43.028 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.028 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.029 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.029 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.029 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.029 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.030 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:08:43.030 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:08:43.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.702+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:43.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.727+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:43.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.727+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:43.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.729+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:43.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.751+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:43.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.760+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:43.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.761+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:43.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.806+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:43.822 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.819+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:43.962 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:43.960+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:46.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.132+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:46.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.132+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:46.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.132+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:46.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.151+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:46.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.203+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:46.205 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.203+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:46.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.647+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:46.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.656+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:46.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.656+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:46.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.656+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:46.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.656+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:46.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:46.698+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:48.563 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.562+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:48.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.562+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:48.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.630+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:48.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.668+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:48.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.679+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:48.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.679+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:48.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.714+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:48.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.755+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:48.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:48.758+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:49.209 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.199+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:49.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.307+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:49.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.324+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:49.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.338+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:49.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.338+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:49.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.349+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:49.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.351+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:49.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.351+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:49.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.585+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:49.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.615+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:49.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.678+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:49.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.678+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:49.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.726+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:49.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:49.726+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:50.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:50.107+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:50.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:50.107+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:50.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:50.107+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:51.418 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:08:51.416+0000 7f5801100700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T18:08:58.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.002+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:08:58.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.002+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:58.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.002+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:58.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.009+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:58.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.137+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:58.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.152+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:58.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.166+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:58.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.166+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:58.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.193+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:58.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.276+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:58.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:58.276+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:08:59.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.434+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:08:59.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.497+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:08:59.509 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.504+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:08:59.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.504+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:08:59.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.591+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:08:59.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.662+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:08:59.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.662+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:08:59.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.670+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:08:59.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.688+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:08:59.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:08:59.725+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:01.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:01.572+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:01.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:01.754+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:01.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:01.958+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:02.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.032+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:02.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.032+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:02.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.104+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:02.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:02.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.229+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:02.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.660+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:02.693 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.691+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:02.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.749+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:02.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.749+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:02.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.749+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:02.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.861+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:02.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:02.861+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:03.031 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:09:03.703 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:09:03.750 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:09:03.758 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1209,"stamp":"2023-10-03T18:09:02.707265+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22491694947,"num_objects":34417,"num_object_clones":29150,"num_object_copies":68834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34417,"num_whiteouts":0,"num_read":1039113,"num_read_kb":3879894,"num_write":981830,"num_write_kb":3924057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80477,"ondisk_log_size":80477,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11849908,"kb_used_data":7845848,"kb_used_omap":3232,"kb_used_meta":4000799,"kb_avail":271265612,"statfs":{"total":289910292480,"available":277775986688,"internally_reserved":0,"allocated":8034148352,"data_stored":48796078441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":4096819151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,15,12,16,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6590514,"num_objects":-193,"num_object_clones":-194,"num_object_copies":-386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-193,"num_whiteouts":0,"num_read":2940,"num_read_kb":7366,"num_write":1606,"num_write_kb":6378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.024559"},"pg_stats":[{"pgid":"2.7","version":"126'132821","reported_seq":253391,"reported_epoch":126,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:08:58.002212+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:08:58.002212+0000","last_peered":"2023-10-03T18:08:58.002212+0000","last_clean":"2023-10-03T18:08:58.002212+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:08:58.002212+0000","last_undegraded":"2023-10-03T18:08:58.002212+0000","last_fullsized":"2023-10-03T18:08:58.002212+0000","mapping_epoch":12,"log_start":"105'122749","ondisk_log_start":"105'122749","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2958,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 440s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2803380709,"num_objects":5639,"num_object_clones":4988,"num_object_copies":11278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":127826,"num_read_kb":483409,"num_write":122249,"num_write_kb":488572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"126'122569","reported_seq":233943,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:57.861087+0000","last_change":"2023-10-03T18:08:57.043048+0000","last_active":"2023-10-03T18:08:57.861087+0000","last_peered":"2023-10-03T18:08:57.861087+0000","last_clean":"2023-10-03T18:08:57.861087+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:08:57.861087+0000","last_undegraded":"2023-10-03T18:08:57.861087+0000","last_fullsized":"2023-10-03T18:08:57.861087+0000","mapping_epoch":12,"log_start":"105'112497","ondisk_log_start":"105'112497","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":21,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":341,"snaptrim_duration":48.202632852999997,"stat_sum":{"num_bytes":2589028974,"num_objects":3438,"num_object_clones":2834,"num_object_copies":6876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3438,"num_whiteouts":0,"num_read":119416,"num_read_kb":446970,"num_write":112806,"num_write_kb":450772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"}]},{"pgid":"2.5","version":"126'147684","reported_seq":278460,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:57.804682+0000","last_change":"2023-10-03T18:08:57.121051+0000","last_active":"2023-10-03T18:08:57.804682+0000","last_peered":"2023-10-03T18:08:57.804682+0000","last_clean":"2023-10-03T18:08:57.804682+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:08:57.804682+0000","last_undegraded":"2023-10-03T18:08:57.804682+0000","last_fullsized":"2023-10-03T18:08:57.804682+0000","mapping_epoch":12,"log_start":"112'137681","ondisk_log_start":"112'137681","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":355,"snaptrim_duration":50.212059859,"stat_sum":{"num_bytes":3034911322,"num_objects":4323,"num_object_clones":3614,"num_object_copies":8646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4323,"num_whiteouts":0,"num_read":142021,"num_read_kb":524281,"num_write":132337,"num_write_kb":528615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"}]},{"pgid":"2.0","version":"127'139693","reported_seq":264694,"reported_epoch":127,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:09:01.449648+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:09:01.449648+0000","last_peered":"2023-10-03T18:09:01.449648+0000","last_clean":"2023-10-03T18:09:01.449648+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:09:01.449648+0000","last_undegraded":"2023-10-03T18:09:01.449648+0000","last_fullsized":"2023-10-03T18:09:01.449648+0000","mapping_epoch":12,"log_start":"112'129595","ondisk_log_start":"112'129595","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":655,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":20,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 77s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2933789429,"num_objects":3876,"num_object_clones":3189,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3876,"num_whiteouts":0,"num_read":133440,"num_read_kb":504507,"num_write":128424,"num_write_kb":513229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":281,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-03T18:08:05.875075+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:08:05.875075+0000","last_peered":"2023-10-03T18:08:05.875075+0000","last_clean":"2023-10-03T18:08:05.875075+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:08:05.875075+0000","last_undegraded":"2023-10-03T18:08:05.875075+0000","last_fullsized":"2023-10-03T18:08:05.875075+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"126'142035","reported_seq":269377,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:57.874919+0000","last_change":"2023-10-03T18:08:57.121176+0000","last_active":"2023-10-03T18:08:57.874919+0000","last_peered":"2023-10-03T18:08:57.874919+0000","last_clean":"2023-10-03T18:08:57.874919+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:08:57.874919+0000","last_undegraded":"2023-10-03T18:08:57.874919+0000","last_fullsized":"2023-10-03T18:08:57.874919+0000","mapping_epoch":12,"log_start":"112'131997","ondisk_log_start":"112'131997","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":27,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":340,"snaptrim_duration":48.372367463000003,"stat_sum":{"num_bytes":2982216625,"num_objects":4132,"num_object_clones":3431,"num_object_copies":8264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4132,"num_whiteouts":0,"num_read":136156,"num_read_kb":514350,"num_write":129835,"num_write_kb":518825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"3"}]},{"pgid":"2.1","version":"127'128870","reported_seq":245731,"reported_epoch":127,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:01.429824+0000","last_change":"2023-10-03T18:09:01.114730+0000","last_active":"2023-10-03T18:09:01.429824+0000","last_peered":"2023-10-03T18:09:01.429824+0000","last_clean":"2023-10-03T18:09:01.429824+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:09:01.429824+0000","last_undegraded":"2023-10-03T18:09:01.429824+0000","last_fullsized":"2023-10-03T18:09:01.429824+0000","mapping_epoch":12,"log_start":"108'118856","ondisk_log_start":"108'118856","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":71,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:07:15.438633+0000","scrub_duration":424.08306342600002,"objects_trimmed":10,"snaptrim_duration":0.40482982099999998,"stat_sum":{"num_bytes":2718269961,"num_objects":5116,"num_object_clones":4483,"num_object_copies":10232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5116,"num_whiteouts":0,"num_read":125538,"num_read_kb":466720,"num_write":118562,"num_write_kb":473607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"9"}]},{"pgid":"2.2","version":"126'130457","reported_seq":250073,"reported_epoch":126,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:08:57.997137+0000","last_change":"2023-10-03T18:08:57.531999+0000","last_active":"2023-10-03T18:08:57.997137+0000","last_peered":"2023-10-03T18:08:57.997137+0000","last_clean":"2023-10-03T18:08:57.997137+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:08:57.997137+0000","last_undegraded":"2023-10-03T18:08:57.997137+0000","last_fullsized":"2023-10-03T18:08:57.997137+0000","mapping_epoch":12,"log_start":"112'120385","ondisk_log_start":"112'120385","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:28:31.732949+0000","scrub_duration":370.574046178,"objects_trimmed":849,"snaptrim_duration":50.690537620000001,"stat_sum":{"num_bytes":2721186565,"num_objects":4247,"num_object_clones":3605,"num_object_copies":8494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":130165,"num_read_kb":473607,"num_write":119512,"num_write_kb":477162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"8"}]},{"pgid":"2.4","version":"126'129606","reported_seq":246108,"reported_epoch":126,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:08:57.870291+0000","last_change":"2023-10-03T18:08:57.043001+0000","last_active":"2023-10-03T18:08:57.870291+0000","last_peered":"2023-10-03T18:08:57.870291+0000","last_clean":"2023-10-03T18:08:57.870291+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:08:57.870291+0000","last_undegraded":"2023-10-03T18:08:57.870291+0000","last_fullsized":"2023-10-03T18:08:57.870291+0000","mapping_epoch":12,"log_start":"112'119597","ondisk_log_start":"112'119597","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":349,"snaptrim_duration":50.192325128999997,"stat_sum":{"num_bytes":2708320994,"num_objects":3644,"num_object_clones":3006,"num_object_copies":7288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3644,"num_whiteouts":0,"num_read":124445,"num_read_kb":465959,"num_write":117968,"num_write_kb":471381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22491104579,"num_objects":34415,"num_object_clones":29150,"num_object_copies":68830,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34415,"num_whiteouts":0,"num_read":1039007,"num_read_kb":3879803,"num_write":981693,"num_write_kb":3922163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8030806016,"data_stored":48775047864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80378,"ondisk_log_size":80378,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2215004,"kb_used_data":1506832,"kb_used_omap":969,"kb_used_meta":707190,"kb_avail":92156836,"statfs":{"total":96636764160,"available":94368600064,"internally_reserved":0,"allocated":1542995968,"data_stored":9279931922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":724163438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,6,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.7440000000000002,"5min":2.589,"15min":2.4860000000000002},"min":{"1min":0.251,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":32.173000000000002},"last":11.462},{"interface":"front","average":{"1min":1.9710000000000001,"5min":2.2989999999999999,"15min":2.7330000000000001},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.17199999999999999},"max":{"1min":12.054,"5min":16.285,"15min":18.041},"last":4.5069999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.5019999999999998,"5min":2.6909999999999998,"15min":2.548},"min":{"1min":0.36199999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.460000000000001,"5min":13.945,"15min":26.074000000000002},"last":15.08},{"interface":"front","average":{"1min":2.7269999999999999,"5min":2.2519999999999998,"15min":2.1040000000000001},"min":{"1min":0.26000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.436999999999999,"5min":19.097000000000001,"15min":19.097000000000001},"last":14.917}]}]},{"osd":0,"up_from":8,"seq":34359738803,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3876924,"kb_used_data":2417588,"kb_used_omap":347,"kb_used_meta":1458980,"kb_avail":90494916,"statfs":{"total":96636764160,"available":92666793984,"internally_reserved":0,"allocated":2475610112,"data_stored":15095994733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1493995696},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,2,2,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":3.3839999999999999,"15min":3.798},"min":{"1min":0.215,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":9.3360000000000003,"5min":30.852,"15min":30.852},"last":0.42299999999999999},{"interface":"front","average":{"1min":2.1579999999999999,"5min":2.8090000000000002,"15min":3.3450000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.19400000000000001},"max":{"1min":12.077999999999999,"5min":16.584,"15min":24.215},"last":0.186}]},{"osd":2,"last update":"Tue Oct 3 18:08:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.607,"15min":2.145},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":8.2609999999999992,"5min":11.818,"15min":14.195},"last":6.2649999999999997},{"interface":"front","average":{"1min":2.028,"5min":2.085,"15min":2.2799999999999998},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.031000000000001,"5min":12.042999999999999,"15min":12.630000000000001},"last":12.119999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738793,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5757980,"kb_used_data":3921428,"kb_used_omap":1915,"kb_used_meta":1834628,"kb_avail":88613860,"statfs":{"total":96636764160,"available":90740592640,"internally_reserved":0,"allocated":4015542272,"data_stored":24420151786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":1878660017},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,6,9,8,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":2.0379999999999998,"5min":2.5720000000000001,"15min":3.1619999999999999},"min":{"1min":0.255,"5min":0.16300000000000001,"15min":0.16300000000000001},"max":{"1min":11.467000000000001,"5min":16.997,"15min":16.997},"last":4.1310000000000002},{"interface":"front","average":{"1min":2.7480000000000002,"5min":3.097,"15min":3.5209999999999999},"min":{"1min":0.19,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":9.9510000000000005,"5min":20.033000000000001,"15min":24.481000000000002},"last":4.0890000000000004}]},{"osd":2,"last update":"Tue Oct 3 18:07:54 2023","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":2.5190000000000001,"15min":2.7120000000000002},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":3.726,"5min":17.492999999999999,"15min":24.579000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.133,"5min":2.649,"15min":2.8340000000000001},"min":{"1min":0.245,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.890000000000001,"5min":20.108000000000001,"15min":22.739000000000001},"last":1.2290000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2474078208,"data_stored":15093750932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4014571520,"data_stored":24402703633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1542156288,"data_stored":9278593299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:09:03.760 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.760 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.761 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.761 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.761 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.761 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:03.761 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:09:03.911 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:03.912+0000 7fc515f98700 -1 reset not still connected to 0x562219557860 2023-10-03T18:09:03.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:03.944+0000 7fc515f98700 -1 reset not still connected to 0x562230045860 2023-10-03T18:09:09.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.718+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:09.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.770+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:09.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.819+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:09.820 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.819+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:09.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.862+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:09.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.917+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:09.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.936+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:09.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.954+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:09.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:09.995+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:10.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:10.089+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:10.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:10.089+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:11.521 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.515+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:09:11.525 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.524+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:09:11.530 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.525+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:09:11.530 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.528+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:09:11.677 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.677+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:09:11.702 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:11.700+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:09:11.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:11.897+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:11.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:11.972+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:12.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.116+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:12.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.136+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:12.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.330+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:12.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.331+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:12.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.350+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:12.388 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.378+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:12.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.642+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:12.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.654+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:12.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.665+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:12.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.740+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:12.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.800+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:12.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.829+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:12.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:12.902 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.902+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:12.902 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:12.902+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:13.050 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:13.047+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:13.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:13.141+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:13.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:13.202+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:17.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.080+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:17.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.098+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:17.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.105+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:17.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.173+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:17.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.193+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:17.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.193+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:17.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.193+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:17.203 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.203+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:17.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.281+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:17.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.716+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:17.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.716+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:17.776 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.776+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:17.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.785+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:17.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.799+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:17.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.799+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:17.837 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.837+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:17.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.934+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:17.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:17.935+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:18.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.363+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:18.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.509+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:18.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.548+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:18.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.548+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:18.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.897+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:18.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.897+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:18.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.897+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:18.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.927+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:18.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.927+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:18.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:18.989+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:19.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.002+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:19.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.004+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:19.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.004+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:19.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.018+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:19.019 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.018+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:19.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.105+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:19.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.129+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:19.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.129+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:19.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.170+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:19.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.197+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:19.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.219+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:19.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.219+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:19.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.220+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:19.233 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.233+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:19.261 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.256+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:19.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.324+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:19.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.324+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:19.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.326+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:19.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.326+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:19.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.367+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:19.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.855+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:19.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.896+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:19.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:19.938+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:20.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:20.109+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:20.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:20.109+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:20.123 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:20.123+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:20.123 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:20.123+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:20.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:20.166+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:23.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:09:24.505 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:09:24.507 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:09:24.525 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1223,"stamp":"2023-10-03T18:09:22.700373+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22512703281,"num_objects":36049,"num_object_clones":30783,"num_object_copies":72098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36049,"num_whiteouts":0,"num_read":1047141,"num_read_kb":3901796,"num_write":987120,"num_write_kb":3944543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80573,"ondisk_log_size":80573,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11688576,"kb_used_data":7886568,"kb_used_omap":3232,"kb_used_meta":3798751,"kb_avail":271426944,"statfs":{"total":289910292480,"available":277941190656,"internally_reserved":0,"allocated":8075845632,"data_stored":51382714239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":3889921999},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,3,0,8,9,17,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":41,"apply_latency_ms":41,"commit_latency_ns":41000000,"apply_latency_ns":41000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11571200,"num_objects":1087,"num_object_clones":1087,"num_object_copies":2174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1087,"num_whiteouts":0,"num_read":2753,"num_read_kb":10794,"num_write":2809,"num_write_kb":11225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.010733"},"pg_stats":[{"pgid":"2.7","version":"133'134014","reported_seq":254947,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:09:20.407104+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:09:20.407104+0000","last_peered":"2023-10-03T18:09:20.407104+0000","last_clean":"2023-10-03T18:09:20.407104+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:09:20.407104+0000","last_undegraded":"2023-10-03T18:09:20.407104+0000","last_fullsized":"2023-10-03T18:09:20.407104+0000","mapping_epoch":12,"log_start":"112'123951","ondisk_log_start":"112'123951","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3122,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":103,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 463s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2806096510,"num_objects":5988,"num_object_clones":5334,"num_object_copies":11976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5988,"num_whiteouts":0,"num_read":128730,"num_read_kb":486035,"num_write":122926,"num_write_kb":491199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'123846","reported_seq":235597,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:20.408861+0000","last_change":"2023-10-03T18:09:18.577287+0000","last_active":"2023-10-03T18:09:20.408861+0000","last_peered":"2023-10-03T18:09:20.408861+0000","last_clean":"2023-10-03T18:09:20.408861+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:09:20.408861+0000","last_undegraded":"2023-10-03T18:09:20.408861+0000","last_fullsized":"2023-10-03T18:09:20.408861+0000","mapping_epoch":12,"log_start":"112'113802","ondisk_log_start":"112'113802","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":79,"snaptrim_duration":6.4559960859999999,"stat_sum":{"num_bytes":2591732434,"num_objects":3660,"num_object_clones":3054,"num_object_copies":7320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3660,"num_whiteouts":0,"num_read":120314,"num_read_kb":449594,"num_write":113481,"num_write_kb":453423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"3"}]},{"pgid":"2.5","version":"133'149163","reported_seq":280378,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:20.352249+0000","last_change":"2023-10-03T18:09:18.675080+0000","last_active":"2023-10-03T18:09:20.352249+0000","last_peered":"2023-10-03T18:09:20.352249+0000","last_clean":"2023-10-03T18:09:20.352249+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:09:20.352249+0000","last_undegraded":"2023-10-03T18:09:20.352249+0000","last_fullsized":"2023-10-03T18:09:20.352249+0000","mapping_epoch":12,"log_start":"112'139086","ondisk_log_start":"112'139086","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":74,"snaptrim_duration":9.0007568669999998,"stat_sum":{"num_bytes":3037950454,"num_objects":4574,"num_object_clones":3867,"num_object_copies":9148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4574,"num_whiteouts":0,"num_read":143378,"num_read_kb":527567,"num_write":133113,"num_write_kb":531588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"3"}]},{"pgid":"2.0","version":"133'140840","reported_seq":266119,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:09:21.563310+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:09:21.563310+0000","last_peered":"2023-10-03T18:09:21.563310+0000","last_clean":"2023-10-03T18:09:21.563310+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:09:21.563310+0000","last_undegraded":"2023-10-03T18:09:21.563310+0000","last_fullsized":"2023-10-03T18:09:21.563310+0000","mapping_epoch":12,"log_start":"112'130798","ondisk_log_start":"112'130798","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":846,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 97s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2936275599,"num_objects":4244,"num_object_clones":3559,"num_object_copies":8488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4244,"num_whiteouts":0,"num_read":134176,"num_read_kb":506938,"num_write":129038,"num_write_kb":515637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":288,"reported_epoch":132,"state":"active+clean","last_fresh":"2023-10-03T18:09:05.755716+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:09:05.755716+0000","last_peered":"2023-10-03T18:09:05.755716+0000","last_clean":"2023-10-03T18:09:05.755716+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:09:05.755716+0000","last_undegraded":"2023-10-03T18:09:05.755716+0000","last_fullsized":"2023-10-03T18:09:05.755716+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'143401","reported_seq":271112,"reported_epoch":133,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:09:20.288549+0000","last_change":"2023-10-03T18:09:18.655289+0000","last_active":"2023-10-03T18:09:20.288549+0000","last_peered":"2023-10-03T18:09:20.288549+0000","last_clean":"2023-10-03T18:09:20.288549+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:09:20.288549+0000","last_undegraded":"2023-10-03T18:09:20.288549+0000","last_fullsized":"2023-10-03T18:09:20.288549+0000","mapping_epoch":12,"log_start":"112'133304","ondisk_log_start":"112'133304","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":70,"snaptrim_duration":11.887668633000001,"stat_sum":{"num_bytes":2985079574,"num_objects":4384,"num_object_clones":3686,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":137154,"num_read_kb":517307,"num_write":130552,"num_write_kb":521609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"}]},{"pgid":"2.1","version":"133'130428","reported_seq":247497,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:21.567292+0000","last_change":"2023-10-03T18:09:19.812892+0000","last_active":"2023-10-03T18:09:21.567292+0000","last_peered":"2023-10-03T18:09:21.567292+0000","last_clean":"2023-10-03T18:09:21.567292+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:09:21.567292+0000","last_undegraded":"2023-10-03T18:09:21.567292+0000","last_fullsized":"2023-10-03T18:09:21.567292+0000","mapping_epoch":12,"log_start":"112'120367","ondisk_log_start":"112'120367","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":81,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:54:25.060172+0000","scrub_duration":424.08306342600002,"objects_trimmed":245,"snaptrim_duration":13.070044477,"stat_sum":{"num_bytes":2720510575,"num_objects":5121,"num_object_clones":4486,"num_object_copies":10242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5121,"num_whiteouts":0,"num_read":126389,"num_read_kb":469144,"num_write":119132,"num_write_kb":475812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"7"}]},{"pgid":"2.2","version":"133'131980","reported_seq":251967,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:18.125159+0000","last_change":"2023-10-03T18:09:17.583182+0000","last_active":"2023-10-03T18:09:18.125159+0000","last_peered":"2023-10-03T18:09:18.125159+0000","last_clean":"2023-10-03T18:09:18.125159+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:09:18.125159+0000","last_undegraded":"2023-10-03T18:09:18.125159+0000","last_fullsized":"2023-10-03T18:09:18.125159+0000","mapping_epoch":12,"log_start":"112'121894","ondisk_log_start":"112'121894","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":51,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:11.026254+0000","scrub_duration":370.574046178,"objects_trimmed":172,"snaptrim_duration":10.739728642999999,"stat_sum":{"num_bytes":2723373830,"num_objects":4209,"num_object_clones":3567,"num_object_copies":8418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4209,"num_whiteouts":0,"num_read":131423,"num_read_kb":476386,"num_write":120086,"num_write_kb":479328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"9"}]},{"pgid":"2.4","version":"133'130907","reported_seq":247787,"reported_epoch":133,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:09:20.305454+0000","last_change":"2023-10-03T18:09:18.561114+0000","last_active":"2023-10-03T18:09:20.305454+0000","last_peered":"2023-10-03T18:09:20.305454+0000","last_clean":"2023-10-03T18:09:20.305454+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:09:20.305454+0000","last_undegraded":"2023-10-03T18:09:20.305454+0000","last_fullsized":"2023-10-03T18:09:20.305454+0000","mapping_epoch":12,"log_start":"112'120903","ondisk_log_start":"112'120903","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":66,"snaptrim_duration":11.783221018000001,"stat_sum":{"num_bytes":2711093937,"num_objects":3867,"num_object_clones":3230,"num_object_copies":7734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3867,"num_whiteouts":0,"num_read":125471,"num_read_kb":468734,"num_write":118655,"num_write_kb":474053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22512112913,"num_objects":36047,"num_object_clones":30783,"num_object_copies":72094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36047,"num_whiteouts":0,"num_read":1047035,"num_read_kb":3901705,"num_write":986983,"num_write_kb":3942649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8071884800,"data_stored":51417754146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80474,"ondisk_log_size":80474,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1847696,"kb_used_data":1513864,"kb_used_omap":969,"kb_used_meta":332854,"kb_avail":92524144,"statfs":{"total":96636764160,"available":94744723456,"internally_reserved":0,"allocated":1550196736,"data_stored":9829402739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":340843374},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,1,2,2,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.7440000000000002,"5min":2.589,"15min":2.4860000000000002},"min":{"1min":0.251,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.254},{"interface":"front","average":{"1min":1.9710000000000001,"5min":2.2989999999999999,"15min":2.7330000000000001},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.17199999999999999},"max":{"1min":12.054,"5min":16.285,"15min":18.041},"last":0.21199999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.5019999999999998,"5min":2.6909999999999998,"15min":2.548},"min":{"1min":0.36199999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.460000000000001,"5min":13.945,"15min":26.074000000000002},"last":0.247},{"interface":"front","average":{"1min":2.7269999999999999,"5min":2.2519999999999998,"15min":2.1040000000000001},"min":{"1min":0.26000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.436999999999999,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.27600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738807,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3938808,"kb_used_data":2429488,"kb_used_omap":347,"kb_used_meta":1508964,"kb_avail":90433032,"statfs":{"total":96636764160,"available":92603424768,"internally_reserved":0,"allocated":2487795712,"data_stored":15840780156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1545179312},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,0,1,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.8639999999999999,"15min":3.7269999999999999},"min":{"1min":0.191,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":8.0790000000000006,"5min":15.081,"15min":30.852},"last":13.180999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":2.5430000000000001,"15min":3.0609999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.5430000000000001,"5min":12.289999999999999,"15min":24.215},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":2.3969999999999998,"5min":1.6180000000000001,"15min":2.1539999999999999},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":9.5860000000000003,"5min":9.5860000000000003,"15min":14.195},"last":12.228999999999999},{"interface":"front","average":{"1min":1.6990000000000001,"5min":2.0350000000000001,"15min":2.274},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.119999999999999,"5min":12.119999999999999,"15min":12.630000000000001},"last":10.823}]}]},{"osd":1,"up_from":8,"seq":34359738797,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5902072,"kb_used_data":3943216,"kb_used_omap":1915,"kb_used_meta":1956932,"kb_avail":88469768,"statfs":{"total":96636764160,"available":90593042432,"internally_reserved":0,"allocated":4037853184,"data_stored":25712531344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":2003899313},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,0,7,7,14,7,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:08:59 2023","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.738,"15min":3.0110000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":13.519,"5min":16.997,"15min":16.997},"last":0.39700000000000002},{"interface":"front","average":{"1min":2.516,"5min":2.7930000000000001,"15min":3.468},"min":{"1min":0.23899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":10.5,"5min":20.033000000000001,"15min":24.481000000000002},"last":0.253}]},{"osd":2,"last update":"Tue Oct 3 18:08:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":2.4049999999999998,"15min":2.766},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":16.263999999999999,"5min":16.263999999999999,"15min":24.579000000000001},"last":3.4820000000000002},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.5419999999999998,"15min":2.8260000000000001},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.557,"5min":20.108000000000001,"15min":22.739000000000001},"last":1.7070000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2486226944,"data_stored":15851814141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4036395008,"data_stored":25733371153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1549262848,"data_stored":9832568852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:09:24.527 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.527 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.527 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.527 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.528 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.528 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:24.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:09:29.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:29.291+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:29.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:29.324+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:29.339 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:29.337+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:29.339 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:29.337+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:31.983 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:31.982+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:31.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:31.982+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:31.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:31.982+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:31.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:31.982+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:32.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.041+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:32.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.236+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:32.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.236+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:32.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.236+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:32.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.268+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:32.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.349+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:32.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.349+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:32.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.639+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:32.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.735+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:32.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.735+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:32.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.735+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:32.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.735+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:32.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.800+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:32.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:32.800+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:33.794 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:33.787+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:33.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:33.882+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:33.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:33.882+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:33.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:33.921+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:33.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:33.958+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:34.226 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:34.225+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:09:34.226 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:34.225+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:09:34.226 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:34.225+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:09:34.538 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:34.537+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:09:37.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:37.333+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:37.418 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:37.413+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:37.418 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:37.413+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:37.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:37.435+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:37.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:37.435+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:39.974 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:39.968+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:39.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:39.990+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:39.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:39.990+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:39.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:39.991+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:40.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.000+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:40.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.061+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:40.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.061+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:40.071 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.070+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:40.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.114+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:40.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:40.226+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:42.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.911+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:42.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.918+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:42.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.918+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:42.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.918+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:42.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.918+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:42.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.954+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:42.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:42.968+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:43.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.064+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:43.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.178+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:43.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.178+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:43.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.210+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:43.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.223+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:43.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.224+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:43.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.894+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:43.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.931+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:43.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.931+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:43.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:43.931+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:44.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:44.065+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:44.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:44.065+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:44.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:44.065+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:44.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:44.076+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:44.529 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:09:45.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:09:45.241 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:09:45.257 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1234,"stamp":"2023-10-03T18:09:44.720143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22536845105,"num_objects":36433,"num_object_clones":31167,"num_object_copies":72866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36433,"num_whiteouts":0,"num_read":1052746,"num_read_kb":3923916,"num_write":992957,"num_write_kb":3967891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80426,"ondisk_log_size":80426,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11883240,"kb_used_data":7934412,"kb_used_omap":3232,"kb_used_meta":3945567,"kb_avail":271232280,"statfs":{"total":289910292480,"available":277741854720,"internally_reserved":0,"allocated":8124837888,"data_stored":52275584895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3309617,"internal_metadata":4040261583},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,4,6,13,26,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12865536,"num_objects":-48,"num_object_clones":-48,"num_object_copies":-96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-48,"num_whiteouts":0,"num_read":3072,"num_read_kb":12140,"num_write":3114,"num_write_kb":12456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.008460"},"pg_stats":[{"pgid":"2.7","version":"133'135128","reported_seq":256559,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:09:42.179890+0000","last_change":"2023-10-03T18:01:37.317549+0000","last_active":"2023-10-03T18:09:42.179890+0000","last_peered":"2023-10-03T18:09:42.179890+0000","last_clean":"2023-10-03T18:09:42.179890+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:09:42.179890+0000","last_undegraded":"2023-10-03T18:09:42.179890+0000","last_fullsized":"2023-10-03T18:09:42.179890+0000","mapping_epoch":12,"log_start":"112'125054","ondisk_log_start":"112'125054","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3318,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":103,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 484s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2809074302,"num_objects":6203,"num_object_clones":5549,"num_object_copies":12406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6203,"num_whiteouts":0,"num_read":129428,"num_read_kb":488795,"num_write":123635,"num_write_kb":494035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"133'124973","reported_seq":237133,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:42.190442+0000","last_change":"2023-10-03T18:09:41.219578+0000","last_active":"2023-10-03T18:09:42.190442+0000","last_peered":"2023-10-03T18:09:42.190442+0000","last_clean":"2023-10-03T18:09:42.190442+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:09:42.190442+0000","last_undegraded":"2023-10-03T18:09:42.190442+0000","last_fullsized":"2023-10-03T18:09:42.190442+0000","mapping_epoch":12,"log_start":"112'114912","ondisk_log_start":"112'114912","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":32,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":216,"snaptrim_duration":31.796711857999998,"stat_sum":{"num_bytes":2594501330,"num_objects":3701,"num_object_clones":3095,"num_object_copies":7402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":120950,"num_read_kb":452102,"num_write":114150,"num_write_kb":456099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"8"}]},{"pgid":"2.5","version":"133'150507","reported_seq":282215,"reported_epoch":133,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:09:42.013628+0000","last_change":"2023-10-03T18:09:41.205436+0000","last_active":"2023-10-03T18:09:42.013628+0000","last_peered":"2023-10-03T18:09:42.013628+0000","last_clean":"2023-10-03T18:09:42.013628+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:09:42.013628+0000","last_undegraded":"2023-10-03T18:09:42.013628+0000","last_fullsized":"2023-10-03T18:09:42.013628+0000","mapping_epoch":12,"log_start":"117'140494","ondisk_log_start":"117'140494","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":208,"snaptrim_duration":34.401993472999997,"stat_sum":{"num_bytes":3041333750,"num_objects":4671,"num_object_clones":3964,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4671,"num_whiteouts":0,"num_read":144141,"num_read_kb":530599,"num_write":133934,"num_write_kb":534872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"}]},{"pgid":"2.0","version":"133'141907","reported_seq":267657,"reported_epoch":133,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:09:41.610924+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:09:41.610924+0000","last_peered":"2023-10-03T18:09:41.610924+0000","last_clean":"2023-10-03T18:09:41.610924+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:09:41.610924+0000","last_undegraded":"2023-10-03T18:09:41.610924+0000","last_fullsized":"2023-10-03T18:09:41.610924+0000","mapping_epoch":12,"log_start":"112'131899","ondisk_log_start":"112'131899","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1001,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 117s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2939179663,"num_objects":4454,"num_object_clones":3769,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":134834,"num_read_kb":509538,"num_write":129741,"num_write_kb":518449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":289,"reported_epoch":133,"state":"active+clean","last_fresh":"2023-10-03T18:09:06.818363+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:09:06.818363+0000","last_peered":"2023-10-03T18:09:06.818363+0000","last_clean":"2023-10-03T18:09:06.818363+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:09:06.818363+0000","last_undegraded":"2023-10-03T18:09:06.818363+0000","last_fullsized":"2023-10-03T18:09:06.818363+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"133'144713","reported_seq":272853,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:42.036226+0000","last_change":"2023-10-03T18:09:41.219681+0000","last_active":"2023-10-03T18:09:42.036226+0000","last_peered":"2023-10-03T18:09:42.036226+0000","last_clean":"2023-10-03T18:09:42.036226+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:09:42.036226+0000","last_undegraded":"2023-10-03T18:09:42.036226+0000","last_fullsized":"2023-10-03T18:09:42.036226+0000","mapping_epoch":12,"log_start":"112'134614","ondisk_log_start":"112'134614","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":225,"snaptrim_duration":32.379580963000002,"stat_sum":{"num_bytes":2988241686,"num_objects":4453,"num_object_clones":3755,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":137863,"num_read_kb":520079,"num_write":131318,"num_write_kb":524673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"1"}]},{"pgid":"2.1","version":"133'131979","reported_seq":249428,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:41.621541+0000","last_change":"2023-10-03T18:09:41.325247+0000","last_active":"2023-10-03T18:09:41.621541+0000","last_peered":"2023-10-03T18:09:41.621541+0000","last_clean":"2023-10-03T18:09:41.621541+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:09:41.621541+0000","last_undegraded":"2023-10-03T18:09:41.621541+0000","last_fullsized":"2023-10-03T18:09:41.621541+0000","mapping_epoch":12,"log_start":"112'121974","ondisk_log_start":"112'121974","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":69,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:54:25.060172+0000","scrub_duration":424.08306342600002,"objects_trimmed":579,"snaptrim_duration":34.582644232,"stat_sum":{"num_bytes":2723242607,"num_objects":4988,"num_object_clones":4353,"num_object_copies":9976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4988,"num_whiteouts":0,"num_read":127078,"num_read_kb":471872,"num_write":119790,"num_write_kb":478444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"1"}]},{"pgid":"2.2","version":"133'133852","reported_seq":254242,"reported_epoch":133,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:09:43.345581+0000","last_change":"2023-10-03T18:09:41.716230+0000","last_active":"2023-10-03T18:09:43.345581+0000","last_peered":"2023-10-03T18:09:43.345581+0000","last_clean":"2023-10-03T18:09:43.345581+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:09:43.345581+0000","last_undegraded":"2023-10-03T18:09:43.345581+0000","last_fullsized":"2023-10-03T18:09:43.345581+0000","mapping_epoch":12,"log_start":"119'123805","ondisk_log_start":"119'123805","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:11.026254+0000","scrub_duration":370.574046178,"objects_trimmed":571,"snaptrim_duration":34.872074019000003,"stat_sum":{"num_bytes":2726728454,"num_objects":4055,"num_object_clones":3413,"num_object_copies":8110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":132195,"num_read_kb":479430,"num_write":120907,"num_write_kb":482612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"2"}]},{"pgid":"2.4","version":"133'132131","reported_seq":249439,"reported_epoch":133,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:09:42.034344+0000","last_change":"2023-10-03T18:09:41.206196+0000","last_active":"2023-10-03T18:09:42.034344+0000","last_peered":"2023-10-03T18:09:42.034344+0000","last_clean":"2023-10-03T18:09:42.034344+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:09:42.034344+0000","last_undegraded":"2023-10-03T18:09:42.034344+0000","last_fullsized":"2023-10-03T18:09:42.034344+0000","mapping_epoch":12,"log_start":"112'122111","ondisk_log_start":"112'122111","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":227,"snaptrim_duration":34.428330148000001,"stat_sum":{"num_bytes":2713952945,"num_objects":3906,"num_object_clones":3269,"num_object_copies":7812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3906,"num_whiteouts":0,"num_read":126151,"num_read_kb":471410,"num_write":119345,"num_write_kb":476813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22536254737,"num_objects":36431,"num_object_clones":31167,"num_object_copies":72862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36431,"num_whiteouts":0,"num_read":1052640,"num_read_kb":3923825,"num_write":992820,"num_write_kb":3965997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":8604,"num_omap_keys":107,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8121229312,"data_stored":52256664098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45154,"internal_metadata":0},"log_size":80327,"ondisk_log_size":80327,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738812,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1887788,"kb_used_data":1522020,"kb_used_omap":969,"kb_used_meta":364790,"kb_avail":92484052,"statfs":{"total":96636764160,"available":94703669248,"internally_reserved":0,"allocated":1558548480,"data_stored":9976547443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":373545838},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,2,6,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.7440000000000002,"5min":2.589,"15min":2.4860000000000002},"min":{"1min":0.251,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":32.173000000000002,"5min":32.173000000000002,"15min":32.173000000000002},"last":5.3840000000000003},{"interface":"front","average":{"1min":1.9710000000000001,"5min":2.2989999999999999,"15min":2.7330000000000001},"min":{"1min":0.216,"5min":0.21199999999999999,"15min":0.17199999999999999},"max":{"1min":12.054,"5min":16.285,"15min":18.041},"last":0.70999999999999996}]},{"osd":1,"last update":"Tue Oct 3 18:08:56 2023","interfaces":[{"interface":"back","average":{"1min":3.5019999999999998,"5min":2.6909999999999998,"15min":2.548},"min":{"1min":0.36199999999999999,"5min":0.20399999999999999,"15min":0.19600000000000001},"max":{"1min":12.460000000000001,"5min":13.945,"15min":26.074000000000002},"last":0.753},{"interface":"front","average":{"1min":2.7269999999999999,"5min":2.2519999999999998,"15min":2.1040000000000001},"min":{"1min":0.26000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":12.436999999999999,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.51100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4045688,"kb_used_data":2445936,"kb_used_omap":347,"kb_used_meta":1599396,"kb_avail":90326152,"statfs":{"total":96636764160,"available":92493979648,"internally_reserved":0,"allocated":2504638464,"data_stored":16159494012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1637781680},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.8639999999999999,"15min":3.7269999999999999},"min":{"1min":0.191,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":8.0790000000000006,"5min":15.081,"15min":30.852},"last":0.23499999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":2.5430000000000001,"15min":3.0609999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.5430000000000001,"5min":12.289999999999999,"15min":24.215},"last":0.29899999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":2.3969999999999998,"5min":1.6180000000000001,"15min":2.1539999999999999},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":9.5860000000000003,"5min":9.5860000000000003,"15min":14.195},"last":0.27900000000000003},{"interface":"front","average":{"1min":1.6990000000000001,"5min":2.0350000000000001,"15min":2.274},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.119999999999999,"5min":12.119999999999999,"15min":12.630000000000001},"last":0.248}]}]},{"osd":1,"up_from":8,"seq":34359738801,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5949764,"kb_used_data":3966456,"kb_used_omap":1915,"kb_used_meta":1981380,"kb_avail":88422076,"statfs":{"total":96636764160,"available":90544205824,"internally_reserved":0,"allocated":4061650944,"data_stored":26139543440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1961039,"internal_metadata":2028934065},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,3,4,11,19,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:08:59 2023","interfaces":[{"interface":"back","average":{"1min":2.7799999999999998,"5min":2.738,"15min":3.0110000000000001},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":13.519,"5min":16.997,"15min":16.997},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.516,"5min":2.7930000000000001,"15min":3.468},"min":{"1min":0.23899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":10.5,"5min":20.033000000000001,"15min":24.481000000000002},"last":1.4119999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:08:59 2023","interfaces":[{"interface":"back","average":{"1min":2.6030000000000002,"5min":2.4049999999999998,"15min":2.766},"min":{"1min":0.23799999999999999,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":16.263999999999999,"5min":16.263999999999999,"15min":24.579000000000001},"last":0.38800000000000001},{"interface":"front","average":{"1min":2.2559999999999998,"5min":2.5419999999999998,"15min":2.8260000000000001},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.17299999999999999},"max":{"1min":10.557,"5min":20.108000000000001,"15min":22.739000000000001},"last":1.458}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2503196672,"data_stored":16155487485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4060323840,"data_stored":26131059985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25212,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1557708800,"data_stored":9970116628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:09:45.259 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.259 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.260 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.260 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.260 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.260 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:09:45.261 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:09:45.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.292+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:45.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.292+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:45.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.391+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:45.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.391+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:45.512 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.508+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:45.512 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.508+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:45.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.590+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:45.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.662+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:45.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:45.680+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:46.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:46.955+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:46.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:46.984+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:47.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.000+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:47.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.020+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:47.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.056+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:47.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.056+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:47.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.056+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:47.059 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.056+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:47.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.140+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:47.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:47.140+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:48.527 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:48.523+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:09:48.527 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:48.523+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:09:48.531 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:09:48.529+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:09:49.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:49.940+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:50.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.007+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:50.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.160+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:50.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.160+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:50.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.160+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:50.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.160+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:50.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.160+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:50.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.161+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:50.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.491+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:50.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.541+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:50.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.595+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:50.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.595+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:50.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.928+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:50.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.928+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:50.976 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:50.972+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:51.418 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:09:51.417+0000 7f5801100700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T18:09:51.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:51.944+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:51.965 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:51.962+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:51.966 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:51.962+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:51.974 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:51.971+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:51.998 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:51.997+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:52.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.075+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:52.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.075+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:52.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.081+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:52.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.081+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:52.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.139+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:52.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.636+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:52.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.636+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:52.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.636+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:52.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.706+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:52.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.748+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:52.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.817+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:52.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:52.817+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:53.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.123+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:53.132 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.123+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:53.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.229+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:53.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.241+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:53.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.413+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:53.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.413+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:53.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.413+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:53.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.518+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:53.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.518+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:53.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.600+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:53.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.600+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:53.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.602+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:53.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.620+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:53.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.634+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:53.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.634+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:53.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.634+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:53.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.634+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:53.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.634+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:53.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.704+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:53.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.704+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:53.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.778+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:53.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.778+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:53.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.778+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:53.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.799+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:53.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:53.813+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:54.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.301+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:54.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.307+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:54.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.307+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:54.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.307+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:54.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.378+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:54.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.379+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:54.998 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.996+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:54.999 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:54.996+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:55.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:55.006+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:55.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:55.006+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:09:56.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:56.757+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:58.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.568+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:09:58.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.588+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:09:58.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.602+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:09:58.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.674+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:09:58.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.674+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:09:58.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.720+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:09:58.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.783+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:09:58.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.783+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:09:58.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.803+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:09:58.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.803+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:09:58.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:09:58.803+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:01.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.145+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:01.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.145+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:01.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.282+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:01.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.282+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:01.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.290+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:01.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:01.290+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:05.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:10:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:10:05.936 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:10:05.942 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1247,"stamp":"2023-10-03T18:10:04.724023+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22561769315,"num_objects":35664,"num_object_clones":30397,"num_object_copies":71328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35664,"num_whiteouts":0,"num_read":1060749,"num_read_kb":3948888,"num_write":999075,"num_write_kb":3992063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80344,"ondisk_log_size":80344,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11820284,"kb_used_data":7982540,"kb_used_omap":3235,"kb_used_meta":3834460,"kb_avail":271295236,"statfs":{"total":289910292480,"available":277806321664,"internally_reserved":0,"allocated":8174120960,"data_stored":51127219735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3313372,"internal_metadata":3926487332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,16,14,16,16,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6058034,"num_objects":-97,"num_object_clones":-98,"num_object_copies":-194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-97,"num_whiteouts":0,"num_read":3596,"num_read_kb":7516,"num_write":1569,"num_write_kb":5976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001178"},"pg_stats":[{"pgid":"2.7","version":"136'136200","reported_seq":258377,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:03.871107+0000","last_change":"2023-10-03T18:10:02.702527+0000","last_active":"2023-10-03T18:10:03.871107+0000","last_peered":"2023-10-03T18:10:03.871107+0000","last_clean":"2023-10-03T18:10:03.871107+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:10:03.871107+0000","last_undegraded":"2023-10-03T18:10:03.871107+0000","last_fullsized":"2023-10-03T18:10:03.871107+0000","mapping_epoch":12,"log_start":"112'126154","ondisk_log_start":"112'126154","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":112,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:46:56.705227+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.00078914999999999996,"stat_sum":{"num_bytes":2812211888,"num_objects":6295,"num_object_clones":5640,"num_object_copies":12590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6295,"num_whiteouts":0,"num_read":130446,"num_read_kb":491974,"num_write":124412,"num_write_kb":497104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"136'126059","reported_seq":238897,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:03.839985+0000","last_change":"2023-10-03T18:10:02.509642+0000","last_active":"2023-10-03T18:10:03.839985+0000","last_peered":"2023-10-03T18:10:03.839985+0000","last_clean":"2023-10-03T18:10:03.839985+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:10:03.839985+0000","last_undegraded":"2023-10-03T18:10:03.839985+0000","last_fullsized":"2023-10-03T18:10:03.839985+0000","mapping_epoch":12,"log_start":"114'116017","ondisk_log_start":"114'116017","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":36,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":52.644934028999998,"stat_sum":{"num_bytes":2597257838,"num_objects":3602,"num_object_clones":2998,"num_object_copies":7204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3602,"num_whiteouts":0,"num_read":121906,"num_read_kb":455088,"num_write":114826,"num_write_kb":458759,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"3"}]},{"pgid":"2.5","version":"136'151898","reported_seq":284356,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:03.858411+0000","last_change":"2023-10-03T18:10:02.522581+0000","last_active":"2023-10-03T18:10:03.858411+0000","last_peered":"2023-10-03T18:10:03.858411+0000","last_clean":"2023-10-03T18:10:03.858411+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:10:03.858411+0000","last_undegraded":"2023-10-03T18:10:03.858411+0000","last_fullsized":"2023-10-03T18:10:03.858411+0000","mapping_epoch":12,"log_start":"119'141894","ondisk_log_start":"119'141894","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.73870619500000001,"stat_sum":{"num_bytes":3044901416,"num_objects":4596,"num_object_clones":3888,"num_object_copies":9192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":145410,"num_read_kb":534121,"num_write":134826,"num_write_kb":538372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"3"}]},{"pgid":"2.0","version":"135'142969","reported_seq":269426,"reported_epoch":135,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:10:01.925650+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:10:01.925650+0000","last_peered":"2023-10-03T18:10:01.925650+0000","last_clean":"2023-10-03T18:10:01.925650+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:10:01.925650+0000","last_undegraded":"2023-10-03T18:10:01.925650+0000","last_fullsized":"2023-10-03T18:10:01.925650+0000","mapping_epoch":12,"log_start":"119'132900","ondisk_log_start":"119'132900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1198,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 137s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2942513857,"num_objects":4527,"num_object_clones":3841,"num_object_copies":9054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":135693,"num_read_kb":512515,"num_write":130553,"num_write_kb":521685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":292,"reported_epoch":135,"state":"active+clean","last_fresh":"2023-10-03T18:10:01.753437+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:10:01.753437+0000","last_peered":"2023-10-03T18:10:01.753437+0000","last_clean":"2023-10-03T18:10:01.753437+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:10:01.753437+0000","last_undegraded":"2023-10-03T18:10:01.753437+0000","last_fullsized":"2023-10-03T18:10:01.753437+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"136'146025","reported_seq":274899,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:03.860666+0000","last_change":"2023-10-03T18:10:02.528043+0000","last_active":"2023-10-03T18:10:03.860666+0000","last_peered":"2023-10-03T18:10:03.860666+0000","last_clean":"2023-10-03T18:10:03.860666+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:10:03.860666+0000","last_undegraded":"2023-10-03T18:10:03.860666+0000","last_fullsized":"2023-10-03T18:10:03.860666+0000","mapping_epoch":12,"log_start":"119'136018","ondisk_log_start":"119'136018","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.055573892999999999,"stat_sum":{"num_bytes":2991723236,"num_objects":4384,"num_object_clones":3687,"num_object_copies":8768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4384,"num_whiteouts":0,"num_read":138945,"num_read_kb":523558,"num_write":132170,"num_write_kb":528041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"7"}]},{"pgid":"2.1","version":"135'133517","reported_seq":251478,"reported_epoch":135,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:01.907108+0000","last_change":"2023-10-03T18:10:01.547526+0000","last_active":"2023-10-03T18:10:01.907108+0000","last_peered":"2023-10-03T18:10:01.907108+0000","last_clean":"2023-10-03T18:10:01.907108+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:10:01.907108+0000","last_undegraded":"2023-10-03T18:10:01.907108+0000","last_fullsized":"2023-10-03T18:10:01.907108+0000","mapping_epoch":12,"log_start":"119'123479","ondisk_log_start":"119'123479","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":59,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:18:11.750968+0000","scrub_duration":424.08306342600002,"objects_trimmed":2,"snaptrim_duration":0.83058850900000003,"stat_sum":{"num_bytes":2726085181,"num_objects":4669,"num_object_clones":4035,"num_object_copies":9338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4669,"num_whiteouts":0,"num_read":127784,"num_read_kb":474677,"num_write":120473,"num_write_kb":481172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"4"}]},{"pgid":"2.2","version":"136'135353","reported_seq":256318,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:03.428207+0000","last_change":"2023-10-03T18:10:02.580040+0000","last_active":"2023-10-03T18:10:03.428207+0000","last_peered":"2023-10-03T18:10:03.428207+0000","last_clean":"2023-10-03T18:10:03.428207+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:10:03.428207+0000","last_undegraded":"2023-10-03T18:10:03.428207+0000","last_fullsized":"2023-10-03T18:10:03.428207+0000","mapping_epoch":12,"log_start":"119'125318","ondisk_log_start":"119'125318","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":34,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:03:50.943805+0000","scrub_duration":370.574046178,"objects_trimmed":10,"snaptrim_duration":0.83008768200000005,"stat_sum":{"num_bytes":2729501546,"num_objects":3768,"num_object_clones":3124,"num_object_copies":7536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3768,"num_whiteouts":0,"num_read":133331,"num_read_kb":482344,"num_write":121589,"num_write_kb":485276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"5"}]},{"pgid":"2.4","version":"136'133320","reported_seq":251315,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:03.838944+0000","last_change":"2023-10-03T18:10:02.509626+0000","last_active":"2023-10-03T18:10:03.838944+0000","last_peered":"2023-10-03T18:10:03.838944+0000","last_clean":"2023-10-03T18:10:03.838944+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:10:03.838944+0000","last_undegraded":"2023-10-03T18:10:03.838944+0000","last_fullsized":"2023-10-03T18:10:03.838944+0000","mapping_epoch":12,"log_start":"119'123316","ondisk_log_start":"119'123316","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74074640000000003,"stat_sum":{"num_bytes":2716983985,"num_objects":3821,"num_object_clones":3184,"num_object_copies":7642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3821,"num_whiteouts":0,"num_read":127128,"num_read_kb":474520,"num_write":120089,"num_write_kb":479760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22561178947,"num_objects":35662,"num_object_clones":30397,"num_object_copies":71324,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35662,"num_whiteouts":0,"num_read":1060643,"num_read_kb":3948797,"num_write":998938,"num_write_kb":3990169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8170057728,"data_stored":51167795796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48909,"internal_metadata":0},"log_size":80245,"ondisk_log_size":80245,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738816,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1941708,"kb_used_data":1531516,"kb_used_omap":969,"kb_used_meta":409206,"kb_avail":92430132,"statfs":{"total":96636764160,"available":94648455168,"internally_reserved":0,"allocated":1568272384,"data_stored":9741799053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":419027822},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,4,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":2.3140000000000001,"15min":2.4910000000000001},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":11.462,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.6910000000000001,"5min":2.0299999999999998,"15min":2.6600000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":12.731999999999999,"5min":16.285,"15min":18.041},"last":0.72499999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.9420000000000002,"15min":2.5630000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.19600000000000001},"max":{"1min":15.08,"5min":15.08,"15min":26.074000000000002},"last":5.0570000000000004},{"interface":"front","average":{"1min":2.9380000000000002,"5min":2.5680000000000001,"15min":2.1859999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.188},"max":{"1min":14.917,"5min":19.097000000000001,"15min":19.097000000000001},"last":5.0270000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738816,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4180164,"kb_used_data":2459700,"kb_used_omap":347,"kb_used_meta":1720100,"kb_avail":90191676,"statfs":{"total":96636764160,"available":92356276224,"internally_reserved":0,"allocated":2518732800,"data_stored":15788459564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1761382576},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":1.919,"5min":2.8639999999999999,"15min":3.7269999999999999},"min":{"1min":0.191,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":8.0790000000000006,"5min":15.081,"15min":30.852},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.4850000000000001,"5min":2.5430000000000001,"15min":3.0609999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.5430000000000001,"5min":12.289999999999999,"15min":24.215},"last":2.6659999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:09:14 2023","interfaces":[{"interface":"back","average":{"1min":2.3969999999999998,"5min":1.6180000000000001,"15min":2.1539999999999999},"min":{"1min":0.23000000000000001,"5min":0.215,"15min":0.20100000000000001},"max":{"1min":9.5860000000000003,"5min":9.5860000000000003,"15min":14.195},"last":0.755},{"interface":"front","average":{"1min":1.6990000000000001,"5min":2.0350000000000001,"15min":2.274},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20599999999999999},"max":{"1min":12.119999999999999,"5min":12.119999999999999,"15min":12.630000000000001},"last":0.22800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738805,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5698412,"kb_used_data":3991324,"kb_used_omap":1918,"kb_used_meta":1705153,"kb_avail":88673428,"statfs":{"total":96636764160,"available":90801590272,"internally_reserved":0,"allocated":4087115776,"data_stored":25596961118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1964794,"internal_metadata":1746076934},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,14,9,14,13,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":2.5649999999999999,"15min":2.9159999999999999},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":5.9800000000000004,"5min":16.997,"15min":16.997},"last":2.0710000000000002},{"interface":"front","average":{"1min":1.401,"5min":2.3359999999999999,"15min":3.2999999999999998},"min":{"1min":0.253,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.7809999999999997,"5min":16.405000000000001,"15min":24.481000000000002},"last":2.0990000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.407,"15min":2.7669999999999999},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":9.0830000000000002,"5min":16.263999999999999,"15min":24.579000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.7949999999999999,"5min":2.1600000000000001,"15min":2.5870000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":8.4710000000000001,"5min":10.890000000000001,"15min":22.739000000000001},"last":0.32700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2517377024,"data_stored":15814864225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4085592064,"data_stored":25607484689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28967,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1567088640,"data_stored":9745446882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:10:05.944 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:05.944 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:05.944 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:05.944 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:05.944 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:05.945 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:10:06.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.817+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:06.953 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.948+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:06.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.959+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:06.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.959+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:06.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.960+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:06.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.981+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:06.993 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.992+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:06.993 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:06.992+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:07.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.005+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:07.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.005+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:07.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.051+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:07.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.064+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:07.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.064+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:07.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.064+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:07.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.064+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:07.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.132+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:07.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.192+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:07.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:07.202+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:08.899 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:08.895+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:10:08.899 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:08.898+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:10:08.902 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:08.902+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:10:08.902 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:08.902+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:10:10.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.813+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:10.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.835+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:10.903 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.903+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:10.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.903+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:10.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.903+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:10.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.903+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:10.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:10.903+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:11.071 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.067+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:11.084 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.084+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:11.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.210+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:11.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.252+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:11.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.277+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:11.337 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.337+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:11.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.337+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:11.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.413+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:11.418 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.414+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:11.479 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:11.478+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:15.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.629+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:15.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.700+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:15.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.700+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:15.701 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.700+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:15.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.725+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:15.909 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:15.909+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:16.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:16.053+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:16.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:16.053+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:16.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:16.071+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:16.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:16.247+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:16.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:16.357+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:18.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.508+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:18.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.512+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:18.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.633+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:18.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.633+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:18.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.746+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:18.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.746+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:18.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.746+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:18.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.746+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:18.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:18.746+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:22.479 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:22.471+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:10:22.480 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:22.471+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:10:22.480 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:22.471+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:10:22.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:22.619+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:22.694 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:22.693+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:22.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:22.693+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:23.858 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:23.856+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:23.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:23.932+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:23.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:23.945+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:23.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:23.945+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:23.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:23.945+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:24.272 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:24.271+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:24.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:24.271+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:24.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:24.271+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:24.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:24.282+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:24.339 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:24.339+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:25.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:10:26.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:10:26.596 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:10:26.617 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1262,"stamp":"2023-10-03T18:10:24.734701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22580508465,"num_objects":37277,"num_object_clones":32011,"num_object_copies":74554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37277,"num_whiteouts":0,"num_read":1067352,"num_read_kb":3967606,"num_write":1003736,"num_write_kb":4010287,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80661,"ondisk_log_size":80661,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12053228,"kb_used_data":8019720,"kb_used_omap":3235,"kb_used_meta":4030236,"kb_avail":271062292,"statfs":{"total":289910292480,"available":277567787008,"internally_reserved":0,"allocated":8212193280,"data_stored":53939820777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3313372,"internal_metadata":4126961956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,5,4,8,24,17,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11079680,"num_objects":1037,"num_object_clones":1037,"num_object_copies":2074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1037,"num_whiteouts":0,"num_read":2552,"num_read_kb":10076,"num_write":2692,"num_write_kb":10768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004169"},"pg_stats":[{"pgid":"2.7","version":"140'137235","reported_seq":259623,"reported_epoch":140,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:20.589306+0000","last_change":"2023-10-03T18:10:19.292353+0000","last_active":"2023-10-03T18:10:20.589306+0000","last_peered":"2023-10-03T18:10:20.589306+0000","last_clean":"2023-10-03T18:10:20.589306+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:10:20.589306+0000","last_undegraded":"2023-10-03T18:10:20.589306+0000","last_fullsized":"2023-10-03T18:10:20.589306+0000","mapping_epoch":12,"log_start":"119'127159","ondisk_log_start":"119'127159","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":121,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:39:17.508911+0000","scrub_duration":505.38444223300002,"objects_trimmed":53,"snaptrim_duration":6.5804897069999999,"stat_sum":{"num_bytes":2814460439,"num_objects":6515,"num_object_clones":5863,"num_object_copies":13030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6515,"num_whiteouts":0,"num_read":131102,"num_read_kb":494075,"num_write":124966,"num_write_kb":499272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"}]},{"pgid":"2.6","version":"140'126918","reported_seq":240017,"reported_epoch":140,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:20.551023+0000","last_change":"2023-10-03T18:10:19.141967+0000","last_active":"2023-10-03T18:10:20.551023+0000","last_peered":"2023-10-03T18:10:20.551023+0000","last_clean":"2023-10-03T18:10:20.551023+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:10:20.551023+0000","last_undegraded":"2023-10-03T18:10:20.551023+0000","last_fullsized":"2023-10-03T18:10:20.551023+0000","mapping_epoch":12,"log_start":"119'116820","ondisk_log_start":"119'116820","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":44,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":9.3029346359999998,"stat_sum":{"num_bytes":2599223918,"num_objects":3820,"num_object_clones":3216,"num_object_copies":7640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3820,"num_whiteouts":0,"num_read":122488,"num_read_kb":457046,"num_write":115299,"num_write_kb":460644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"6"}]},{"pgid":"2.5","version":"140'152997","reported_seq":285719,"reported_epoch":140,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:20.480963+0000","last_change":"2023-10-03T18:10:17.259767+0000","last_active":"2023-10-03T18:10:20.480963+0000","last_peered":"2023-10-03T18:10:20.480963+0000","last_clean":"2023-10-03T18:10:20.480963+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:10:20.480963+0000","last_undegraded":"2023-10-03T18:10:20.480963+0000","last_fullsized":"2023-10-03T18:10:20.480963+0000","mapping_epoch":12,"log_start":"119'142897","ondisk_log_start":"119'142897","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":40,"snaptrim_duration":7.3969758859999999,"stat_sum":{"num_bytes":3047277249,"num_objects":4860,"num_object_clones":4149,"num_object_copies":9720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4860,"num_whiteouts":0,"num_read":146306,"num_read_kb":536498,"num_write":135424,"num_write_kb":540697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"5"}]},{"pgid":"2.0","version":"141'144139","reported_seq":270934,"reported_epoch":141,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:10:22.036330+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:10:22.036330+0000","last_peered":"2023-10-03T18:10:22.036330+0000","last_clean":"2023-10-03T18:10:22.036330+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:10:22.036330+0000","last_undegraded":"2023-10-03T18:10:22.036330+0000","last_fullsized":"2023-10-03T18:10:22.036330+0000","mapping_epoch":12,"log_start":"119'134103","ondisk_log_start":"119'134103","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1376,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":60,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 157s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2945168117,"num_objects":4882,"num_object_clones":4195,"num_object_copies":9764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4882,"num_whiteouts":0,"num_read":136591,"num_read_kb":515151,"num_write":131209,"num_write_kb":524250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":296,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-03T18:10:08.796648+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:10:08.796648+0000","last_peered":"2023-10-03T18:10:08.796648+0000","last_clean":"2023-10-03T18:10:08.796648+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:10:08.796648+0000","last_undegraded":"2023-10-03T18:10:08.796648+0000","last_fullsized":"2023-10-03T18:10:08.796648+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"140'147073","reported_seq":276135,"reported_epoch":140,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:20.469299+0000","last_change":"2023-10-03T18:10:19.218908+0000","last_active":"2023-10-03T18:10:20.469299+0000","last_peered":"2023-10-03T18:10:20.469299+0000","last_clean":"2023-10-03T18:10:20.469299+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:10:20.469299+0000","last_undegraded":"2023-10-03T18:10:20.469299+0000","last_fullsized":"2023-10-03T18:10:20.469299+0000","mapping_epoch":12,"log_start":"119'137022","ondisk_log_start":"119'137022","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":51,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":4.6146990070000005,"stat_sum":{"num_bytes":2994045719,"num_objects":4640,"num_object_clones":3942,"num_object_copies":9280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4640,"num_whiteouts":0,"num_read":139592,"num_read_kb":525635,"num_write":132744,"num_write_kb":530305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"9"}]},{"pgid":"2.1","version":"141'135124","reported_seq":253379,"reported_epoch":141,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:22.061595+0000","last_change":"2023-10-03T18:10:19.900691+0000","last_active":"2023-10-03T18:10:22.061595+0000","last_peered":"2023-10-03T18:10:22.061595+0000","last_clean":"2023-10-03T18:10:22.061595+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:10:22.061595+0000","last_undegraded":"2023-10-03T18:10:22.061595+0000","last_fullsized":"2023-10-03T18:10:22.061595+0000","mapping_epoch":12,"log_start":"120'125089","ondisk_log_start":"120'125089","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:18:59.132364+0000","scrub_duration":424.08306342600002,"objects_trimmed":6,"snaptrim_duration":10.077566432999999,"stat_sum":{"num_bytes":2728636989,"num_objects":4717,"num_object_clones":4083,"num_object_copies":9434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4717,"num_whiteouts":0,"num_read":129006,"num_read_kb":477542,"num_write":121126,"num_write_kb":483683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"2"}]},{"pgid":"2.2","version":"141'136995","reported_seq":258236,"reported_epoch":141,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:23.617604+0000","last_change":"2023-10-03T18:10:23.198828+0000","last_active":"2023-10-03T18:10:23.617604+0000","last_peered":"2023-10-03T18:10:23.617604+0000","last_clean":"2023-10-03T18:10:23.617604+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:10:23.617604+0000","last_undegraded":"2023-10-03T18:10:23.617604+0000","last_fullsized":"2023-10-03T18:10:23.617604+0000","mapping_epoch":12,"log_start":"126'126929","ondisk_log_start":"126'126929","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:33:16.842148+0000","scrub_duration":370.574046178,"objects_trimmed":30,"snaptrim_duration":1.4925794030000001,"stat_sum":{"num_bytes":2732225335,"num_objects":3822,"num_object_clones":3179,"num_object_copies":7644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3822,"num_whiteouts":0,"num_read":134332,"num_read_kb":485044,"num_write":122272,"num_write_kb":487939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"5"}]},{"pgid":"2.4","version":"140'134221","reported_seq":252463,"reported_epoch":140,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:20.507694+0000","last_change":"2023-10-03T18:10:19.290727+0000","last_active":"2023-10-03T18:10:20.507694+0000","last_peered":"2023-10-03T18:10:20.507694+0000","last_clean":"2023-10-03T18:10:20.507694+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:10:20.507694+0000","last_undegraded":"2023-10-03T18:10:20.507694+0000","last_fullsized":"2023-10-03T18:10:20.507694+0000","mapping_epoch":12,"log_start":"119'124121","ondisk_log_start":"119'124121","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":45,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":49,"snaptrim_duration":9.4730730580000007,"stat_sum":{"num_bytes":2718880331,"num_objects":4019,"num_object_clones":3384,"num_object_copies":8038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4019,"num_whiteouts":0,"num_read":127829,"num_read_kb":476524,"num_write":120559,"num_write_kb":481603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"44","length":"1"},{"start":"46","length":"1"},{"start":"49","length":"1"},{"start":"57","length":"c"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22579918097,"num_objects":37275,"num_object_clones":32011,"num_object_copies":74550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37275,"num_whiteouts":0,"num_read":1067246,"num_read_kb":3967515,"num_write":1003599,"num_write_kb":4008393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8208379904,"data_stored":54002256418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48909,"internal_metadata":0},"log_size":80562,"ondisk_log_size":80562,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1995056,"kb_used_data":1539048,"kb_used_omap":969,"kb_used_meta":455030,"kb_avail":92376784,"statfs":{"total":96636764160,"available":94593826816,"internally_reserved":0,"allocated":1575985152,"data_stored":10302042230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":465951598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,2,6,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":2.3140000000000001,"15min":2.4910000000000001},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":11.462,"5min":32.173000000000002,"15min":32.173000000000002},"last":2.7650000000000001},{"interface":"front","average":{"1min":1.6910000000000001,"5min":2.0299999999999998,"15min":2.6600000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":12.731999999999999,"5min":16.285,"15min":18.041},"last":3.585}]},{"osd":1,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.9420000000000002,"15min":2.5630000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.19600000000000001},"max":{"1min":15.08,"5min":15.08,"15min":26.074000000000002},"last":6.9459999999999997},{"interface":"front","average":{"1min":2.9380000000000002,"5min":2.5680000000000001,"15min":2.1859999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.188},"max":{"1min":14.917,"5min":19.097000000000001,"15min":19.097000000000001},"last":7.2919999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738820,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4224944,"kb_used_data":2472548,"kb_used_omap":347,"kb_used_meta":1752036,"kb_avail":90146896,"statfs":{"total":96636764160,"available":92310421504,"internally_reserved":0,"allocated":2531889152,"data_stored":16733747477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1794085040},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,5,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":4.3700000000000001,"5min":2.875,"15min":3.6360000000000001},"min":{"1min":0.23499999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":13.180999999999999,"5min":15.081,"15min":30.852},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.5129999999999999,"5min":2.1850000000000001,"15min":2.8239999999999998},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":10.225,"5min":12.289999999999999,"15min":18.492000000000001},"last":0.39300000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":1.603,"15min":2.177},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":12.228999999999999,"5min":12.228999999999999,"15min":14.195},"last":0.375},{"interface":"front","average":{"1min":2.415,"5min":1.9470000000000001,"15min":2.1659999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.20599999999999999},"max":{"1min":12.880000000000001,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.224}]}]},{"osd":1,"up_from":8,"seq":34359738808,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5833228,"kb_used_data":4008124,"kb_used_omap":1918,"kb_used_meta":1823169,"kb_avail":88538612,"statfs":{"total":96636764160,"available":90663538688,"internally_reserved":0,"allocated":4104318976,"data_stored":26904031070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1964794,"internal_metadata":1866925318},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,6,13,13,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":2.5649999999999999,"15min":2.9159999999999999},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":5.9800000000000004,"5min":16.997,"15min":16.997},"last":6.835},{"interface":"front","average":{"1min":1.401,"5min":2.3359999999999999,"15min":3.2999999999999998},"min":{"1min":0.253,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.7809999999999997,"5min":16.405000000000001,"15min":24.481000000000002},"last":4.202}]},{"osd":2,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.407,"15min":2.7669999999999999},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":9.0830000000000002,"5min":16.263999999999999,"15min":24.579000000000001},"last":4.5270000000000001},{"interface":"front","average":{"1min":1.7949999999999999,"5min":2.1600000000000001,"15min":2.5870000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":8.4710000000000001,"5min":10.890000000000001,"15min":22.739000000000001},"last":4.2990000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2530504704,"data_stored":16757415112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4102795264,"data_stored":26924143889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28967,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1575079936,"data_stored":10320697417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:10:26.619 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:26.619 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:26.620 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:26.620 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:26.620 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:26.620 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:10:31.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.508+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:31.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.573+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:31.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:31.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:31.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.584+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:31.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.584+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:31.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.654+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:31.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.654+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:31.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.673+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:31.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.674+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:31.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:31.779+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:32.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.269+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:32.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.292+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:32.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.292+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:32.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.388+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:32.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.388+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:32.392 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.388+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:32.392 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.388+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:32.392 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.388+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:32.392 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.391+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:32.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.611+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:32.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.664+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:32.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.664+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:32.669 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.669+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:32.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:32.707+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:33.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.525+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:33.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.534+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:33.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.594+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:33.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.594+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:33.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.594+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:33.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.609+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:33.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.645+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:33.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.645+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:33.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.908+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:33.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.908+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:33.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:33.945+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:34.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:34.164+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:35.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.325+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:35.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.363+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:35.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.363+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:35.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.363+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:35.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.363+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:35.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.420+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:35.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.426+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:35.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:35.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:35.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:35.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:35.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:35.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:35.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:35.785+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:36.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.072+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:36.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.119+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:36.121 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.119+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:36.121 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.119+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:36.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.161+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:36.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.182+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:36.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.714+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:36.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.725+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:36.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.727+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:36.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.802+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:36.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.802+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:36.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.805+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:36.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.834+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:36.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.834+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:36.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.853+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:36.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.867+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:36.902 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:36.894+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:38.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.272+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:38.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.315+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:38.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.320+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:38.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:38.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:38.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:38.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:38.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:38.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:38.389+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:45.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:45.835+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:45.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:45.841+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:45.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:45.841+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:45.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:45.841+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:45.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:45.841+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:46.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.040+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:46.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.041+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:46.178 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.176+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:46.216 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.212+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:46.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.218+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:46.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.218+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:46.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.218+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:46.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.323+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:46.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:46.372+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:46.621 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:10:47.223 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:10:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:10:47.255 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1278,"stamp":"2023-10-03T18:10:46.747554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22605670193,"num_objects":37750,"num_object_clones":32484,"num_object_copies":75500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37750,"num_whiteouts":0,"num_read":1072926,"num_read_kb":3989678,"num_write":1009840,"num_write_kb":4034703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80479,"ondisk_log_size":80479,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12471796,"kb_used_data":8068312,"kb_used_omap":3235,"kb_used_meta":4400220,"kb_avail":270643724,"statfs":{"total":289910292480,"available":277139173376,"internally_reserved":0,"allocated":8261951488,"data_stored":54803919289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3313372,"internal_metadata":4505825572},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,1,2,5,14,13,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7479296,"num_objects":-73,"num_object_clones":-73,"num_object_copies":-146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-73,"num_whiteouts":0,"num_read":1625,"num_read_kb":6420,"num_write":1812,"num_write_kb":7248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005212"},"pg_stats":[{"pgid":"2.7","version":"145'138475","reported_seq":261321,"reported_epoch":145,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:41.974702+0000","last_change":"2023-10-03T18:10:39.517495+0000","last_active":"2023-10-03T18:10:41.974702+0000","last_peered":"2023-10-03T18:10:41.974702+0000","last_clean":"2023-10-03T18:10:41.974702+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:10:41.974702+0000","last_undegraded":"2023-10-03T18:10:41.974702+0000","last_fullsized":"2023-10-03T18:10:41.974702+0000","mapping_epoch":12,"log_start":"119'128465","ondisk_log_start":"119'128465","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":117,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:39:17.508911+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":2.7946018769999998,"stat_sum":{"num_bytes":2817602071,"num_objects":6619,"num_object_clones":5967,"num_object_copies":13238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6619,"num_whiteouts":0,"num_read":131807,"num_read_kb":496859,"num_write":125730,"num_write_kb":502328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"59","length":"1"}]},{"pgid":"2.6","version":"145'128090","reported_seq":241601,"reported_epoch":145,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:41.901354+0000","last_change":"2023-10-03T18:10:40.843903+0000","last_active":"2023-10-03T18:10:41.901354+0000","last_peered":"2023-10-03T18:10:41.901354+0000","last_clean":"2023-10-03T18:10:41.901354+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:10:41.901354+0000","last_undegraded":"2023-10-03T18:10:41.901354+0000","last_fullsized":"2023-10-03T18:10:41.901354+0000","mapping_epoch":12,"log_start":"119'118028","ondisk_log_start":"119'118028","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.1206722090000003,"stat_sum":{"num_bytes":2602152558,"num_objects":3912,"num_object_clones":3308,"num_object_copies":7824,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3912,"num_whiteouts":0,"num_read":123135,"num_read_kb":459610,"num_write":116013,"num_write_kb":463500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"}]},{"pgid":"2.5","version":"145'154299","reported_seq":287542,"reported_epoch":145,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:10:41.895262+0000","last_change":"2023-10-03T18:10:38.330737+0000","last_active":"2023-10-03T18:10:41.895262+0000","last_peered":"2023-10-03T18:10:41.895262+0000","last_clean":"2023-10-03T18:10:41.895262+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:10:41.895262+0000","last_undegraded":"2023-10-03T18:10:41.895262+0000","last_fullsized":"2023-10-03T18:10:41.895262+0000","mapping_epoch":12,"log_start":"123'144203","ondisk_log_start":"123'144203","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.6030677070000001,"stat_sum":{"num_bytes":3050820289,"num_objects":4993,"num_object_clones":4282,"num_object_copies":9986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":147052,"num_read_kb":539434,"num_write":136284,"num_write_kb":544137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"8"}]},{"pgid":"2.0","version":"145'145266","reported_seq":272561,"reported_epoch":145,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:10:42.376061+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:10:42.376061+0000","last_peered":"2023-10-03T18:10:42.376061+0000","last_clean":"2023-10-03T18:10:42.376061+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:10:42.376061+0000","last_undegraded":"2023-10-03T18:10:42.376061+0000","last_fullsized":"2023-10-03T18:10:42.376061+0000","mapping_epoch":12,"log_start":"119'135204","ondisk_log_start":"119'135204","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1547,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":60,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 177s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2948301557,"num_objects":5089,"num_object_clones":4402,"num_object_copies":10178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5089,"num_whiteouts":0,"num_read":137283,"num_read_kb":517899,"num_write":131966,"num_write_kb":527278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":301,"reported_epoch":144,"state":"active+clean","last_fresh":"2023-10-03T18:10:36.854073+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:10:36.854073+0000","last_peered":"2023-10-03T18:10:36.854073+0000","last_clean":"2023-10-03T18:10:36.854073+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:10:36.854073+0000","last_undegraded":"2023-10-03T18:10:36.854073+0000","last_fullsized":"2023-10-03T18:10:36.854073+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"145'148331","reported_seq":277907,"reported_epoch":145,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:41.815194+0000","last_change":"2023-10-03T18:10:39.530805+0000","last_active":"2023-10-03T18:10:41.815194+0000","last_peered":"2023-10-03T18:10:41.815194+0000","last_clean":"2023-10-03T18:10:41.815194+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:10:41.815194+0000","last_undegraded":"2023-10-03T18:10:41.815194+0000","last_fullsized":"2023-10-03T18:10:41.815194+0000","mapping_epoch":12,"log_start":"119'138324","ondisk_log_start":"119'138324","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.3002226459999999,"stat_sum":{"num_bytes":2997326615,"num_objects":4762,"num_object_clones":4064,"num_object_copies":9524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4762,"num_whiteouts":0,"num_read":140355,"num_read_kb":528667,"num_write":133541,"num_write_kb":533493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"2"}]},{"pgid":"2.1","version":"145'136716","reported_seq":255413,"reported_epoch":145,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:42.375384+0000","last_change":"2023-10-03T18:10:42.060341+0000","last_active":"2023-10-03T18:10:42.375384+0000","last_peered":"2023-10-03T18:10:42.375384+0000","last_clean":"2023-10-03T18:10:42.375384+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:10:42.375384+0000","last_undegraded":"2023-10-03T18:10:42.375384+0000","last_fullsized":"2023-10-03T18:10:42.375384+0000","mapping_epoch":12,"log_start":"126'126696","ondisk_log_start":"126'126696","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:18:59.132364+0000","scrub_duration":424.08306342600002,"objects_trimmed":11,"snaptrim_duration":0.33824016000000001,"stat_sum":{"num_bytes":2731717181,"num_objects":4576,"num_object_clones":3942,"num_object_copies":9152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":129724,"num_read_kb":480390,"num_write":121872,"num_write_kb":486667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"5"}]},{"pgid":"2.2","version":"145'138564","reported_seq":260199,"reported_epoch":145,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:43.970871+0000","last_change":"2023-10-03T18:10:43.712017+0000","last_active":"2023-10-03T18:10:43.970871+0000","last_peered":"2023-10-03T18:10:43.970871+0000","last_clean":"2023-10-03T18:10:43.970871+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:10:43.970871+0000","last_undegraded":"2023-10-03T18:10:43.970871+0000","last_fullsized":"2023-10-03T18:10:43.970871+0000","mapping_epoch":12,"log_start":"126'128536","ondisk_log_start":"126'128536","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:33:16.842148+0000","scrub_duration":370.574046178,"objects_trimmed":38,"snaptrim_duration":1.9333376370000002,"stat_sum":{"num_bytes":2735231799,"num_objects":3667,"num_object_clones":3024,"num_object_copies":7334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3667,"num_whiteouts":0,"num_read":134992,"num_read_kb":487652,"num_write":123004,"num_write_kb":490867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"6"}]},{"pgid":"2.4","version":"145'135420","reported_seq":254060,"reported_epoch":145,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:10:41.901449+0000","last_change":"2023-10-03T18:10:40.882332+0000","last_active":"2023-10-03T18:10:41.901449+0000","last_peered":"2023-10-03T18:10:41.901449+0000","last_clean":"2023-10-03T18:10:41.901449+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:10:41.901449+0000","last_undegraded":"2023-10-03T18:10:41.901449+0000","last_fullsized":"2023-10-03T18:10:41.901449+0000","mapping_epoch":12,"log_start":"119'125325","ondisk_log_start":"119'125325","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":42,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84914627099999995,"stat_sum":{"num_bytes":2721927755,"num_objects":4130,"num_object_clones":3495,"num_object_copies":8260,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4130,"num_whiteouts":0,"num_read":128472,"num_read_kb":479076,"num_write":121293,"num_write_kb":484539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"59","length":"a"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22605079825,"num_objects":37748,"num_object_clones":32484,"num_object_copies":75496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37748,"num_whiteouts":0,"num_read":1072820,"num_read_kb":3989587,"num_write":1009703,"num_write_kb":4032809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8258154496,"data_stored":54788672034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48909,"internal_metadata":0},"log_size":80380,"ondisk_log_size":80380,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2053108,"kb_used_data":1548520,"kb_used_omap":969,"kb_used_meta":503606,"kb_avail":92318732,"statfs":{"total":96636764160,"available":94534381568,"internally_reserved":0,"allocated":1585684480,"data_stored":10480302598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":515693422},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,0,1,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":1.823,"5min":2.3140000000000001,"15min":2.4910000000000001},"min":{"1min":0.19500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":11.462,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.317},{"interface":"front","average":{"1min":1.6910000000000001,"5min":2.0299999999999998,"15min":2.6600000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":12.731999999999999,"5min":16.285,"15min":18.041},"last":0.88900000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:09:59 2023","interfaces":[{"interface":"back","average":{"1min":2.7549999999999999,"5min":2.9420000000000002,"15min":2.5630000000000002},"min":{"1min":0.247,"5min":0.22600000000000001,"15min":0.19600000000000001},"max":{"1min":15.08,"5min":15.08,"15min":26.074000000000002},"last":1.833},{"interface":"front","average":{"1min":2.9380000000000002,"5min":2.5680000000000001,"15min":2.1859999999999999},"min":{"1min":0.246,"5min":0.188,"15min":0.188},"max":{"1min":14.917,"5min":19.097000000000001,"15min":19.097000000000001},"last":1.8839999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738824,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4343620,"kb_used_data":2487100,"kb_used_omap":347,"kb_used_meta":1856164,"kb_avail":90028220,"statfs":{"total":96636764160,"available":92188897280,"internally_reserved":0,"allocated":2546790400,"data_stored":16916709541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1900712112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":4.3700000000000001,"5min":2.875,"15min":3.6360000000000001},"min":{"1min":0.23499999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":13.180999999999999,"5min":15.081,"15min":30.852},"last":1.526},{"interface":"front","average":{"1min":2.5129999999999999,"5min":2.1850000000000001,"15min":2.8239999999999998},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":10.225,"5min":12.289999999999999,"15min":18.492000000000001},"last":0.252}]},{"osd":2,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":1.603,"15min":2.177},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":12.228999999999999,"5min":12.228999999999999,"15min":14.195},"last":0.81599999999999995},{"interface":"front","average":{"1min":2.415,"5min":1.9470000000000001,"15min":2.1659999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.20599999999999999},"max":{"1min":12.880000000000001,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.28799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738812,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6075068,"kb_used_data":4032692,"kb_used_omap":1918,"kb_used_meta":2040449,"kb_avail":88296772,"statfs":{"total":96636764160,"available":90415894528,"internally_reserved":0,"allocated":4129476608,"data_stored":27406907150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1964794,"internal_metadata":2089420038},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,0,4,11,12,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":2.5649999999999999,"15min":2.9159999999999999},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":5.9800000000000004,"5min":16.997,"15min":16.997},"last":3.1120000000000001},{"interface":"front","average":{"1min":1.401,"5min":2.3359999999999999,"15min":3.2999999999999998},"min":{"1min":0.253,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.7809999999999997,"5min":16.405000000000001,"15min":24.481000000000002},"last":0.62}]},{"osd":2,"last update":"Tue Oct 3 18:10:01 2023","interfaces":[{"interface":"back","average":{"1min":2.0609999999999999,"5min":2.407,"15min":2.7669999999999999},"min":{"1min":0.24099999999999999,"5min":0.20100000000000001,"15min":0.17299999999999999},"max":{"1min":9.0830000000000002,"5min":16.263999999999999,"15min":24.579000000000001},"last":5.5670000000000002},{"interface":"front","average":{"1min":1.7949999999999999,"5min":2.1600000000000001,"15min":2.5870000000000002},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":8.4710000000000001,"5min":10.890000000000001,"15min":22.739000000000001},"last":0.66300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2545541120,"data_stored":16905010376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4128227328,"data_stored":27403814161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28967,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1584386048,"data_stored":10479847497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:10:47.256 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:47.256 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:47.257 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:47.257 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:47.257 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:10:47.257 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:10:47.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.551+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:47.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.580+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:47.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.580+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:47.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.594+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:47.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.594+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:47.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.594+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:47.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.594+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:47.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.595+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:47.791 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.786+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:47.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.786+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:47.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.806+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:47.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.847+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:47.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.847+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:47.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.847+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:47.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.847+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:47.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.847+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:47.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.892+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:47.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:47.892+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:48.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.220+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:48.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.351+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:48.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.373+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:48.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.373+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:48.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.373+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:48.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.394+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:48.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:48.530+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:49.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.438+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:49.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.513+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:49.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.513+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:49.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.530+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:49.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.530+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:49.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.592+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:49.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.592+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:49.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.607+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:49.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.607+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:49.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:49.607+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:50.552 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:10:50.549+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:10:50.553 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:10:50.550+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:10:50.553 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:10:50.550+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:10:50.553 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:10:50.550+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:10:51.420 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:10:51.418+0000 7f5801100700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T18:10:52.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.693+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:52.701 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.697+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:52.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.734+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:52.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.734+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:52.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.762+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:52.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.782+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:52.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.844+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:52.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.844+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:52.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.844+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:52.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.872+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:52.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:52.908+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:53.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.580+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:53.585 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:53.585 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:53.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.631+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:53.635 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.633+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:53.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.717+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:53.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.717+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:53.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.717+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:53.768 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.766+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:53.768 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:53.766+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:53.913 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:53.912+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:10:53.913 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:53.912+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:10:53.913 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:53.912+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:10:53.913 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:53.912+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:10:53.914 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:53.912+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:10:55.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.500+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:10:55.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.500+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:55.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.500+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:55.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.500+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:55.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.500+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:55.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.515+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:55.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.515+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:55.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.515+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:55.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.551+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:55.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.551+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:55.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:55.551+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:56.318 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:56.317+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:10:56.319 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:56.317+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:10:56.336 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:56.325+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:10:56.336 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:56.325+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:10:56.336 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:10:56.325+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:10:56.377 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.376+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:56.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.376+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:56.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.388+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:56.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.545+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:56.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.545+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:56.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.545+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:56.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.545+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:56.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.545+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:56.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.656+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:56.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.656+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:56.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.656+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:56.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.682+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:56.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.686+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:56.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:56.865+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:57.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.244+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:57.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.244+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:57.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.247+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:57.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.247+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:57.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.611+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:57.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.618+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:57.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.639+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:57.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.651+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:57.653 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.651+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:57.653 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.651+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:57.768 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.767+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:57.828 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:57.827+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:10:58.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.225+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:10:58.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.249+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:10:58.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.249+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:10:58.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.281+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:10:58.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.281+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:10:58.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.305+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:10:58.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.305+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:10:58.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.305+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:10:58.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.305+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:10:58.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:10:58.402+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:00.479 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.472+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:00.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.514+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:00.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.518+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:00.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.518+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:00.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.533+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:00.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.546+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:00.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.546+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:00.688 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:11:00.685+0000 7f0623ebd700 -1 reset not still connected to 0x55871263fe10 2023-10-03T18:11:00.688 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:11:00.685+0000 7f0623ebd700 -1 reset not still connected to 0x558740b97380 2023-10-03T18:11:00.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.727+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:00.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.727+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:00.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:00.727+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:01.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.268+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:01.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.273+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:01.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.357+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:01.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.357+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:01.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.524+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:01.526 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.526+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:01.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:01.610+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:01.971 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:01.969+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:11:01.972 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:01.969+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:11:02.143 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:02.140+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:11:02.143 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:02.140+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:11:03.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:03.749+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:11:03.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:03.789+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:03.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:03.851+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:03.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:03.851+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:03.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:03.956+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:04.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.000+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:04.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.136+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:04.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.136+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:04.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.136+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:04.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.243+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:04.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:04.303+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:07.258 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:11:07.934 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:11:07.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:11:07.966 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1298,"stamp":"2023-10-03T18:11:06.760216+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22632458135,"num_objects":37144,"num_object_clones":31876,"num_object_copies":74288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37144,"num_whiteouts":0,"num_read":1080941,"num_read_kb":4014745,"num_write":1016436,"num_write_kb":4060669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80618,"ondisk_log_size":80618,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12604368,"kb_used_data":8119852,"kb_used_omap":3235,"kb_used_meta":4481244,"kb_avail":270511152,"statfs":{"total":289910292480,"available":277003419648,"internally_reserved":0,"allocated":8314728448,"data_stored":53906367610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3313372,"internal_metadata":4588794148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,7,13,17,24,8,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5075147,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":3024,"num_read_kb":6095,"num_write":1340,"num_write_kb":4986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000638"},"pg_stats":[{"pgid":"2.7","version":"152'139717","reported_seq":263253,"reported_epoch":152,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:03.614358+0000","last_change":"2023-10-03T18:11:01.588136+0000","last_active":"2023-10-03T18:11:03.614358+0000","last_peered":"2023-10-03T18:11:03.614358+0000","last_clean":"2023-10-03T18:11:03.614358+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:11:03.614358+0000","last_undegraded":"2023-10-03T18:11:03.614358+0000","last_fullsized":"2023-10-03T18:11:03.614358+0000","mapping_epoch":12,"log_start":"119'129668","ondisk_log_start":"119'129668","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":123,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:01:20.483836+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.27829346799999999,"stat_sum":{"num_bytes":2821046907,"num_objects":6595,"num_object_clones":5941,"num_object_copies":13190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6595,"num_whiteouts":0,"num_read":132832,"num_read_kb":500058,"num_write":126594,"num_write_kb":505710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"1"}]},{"pgid":"2.6","version":"152'129200","reported_seq":243351,"reported_epoch":152,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:03.636292+0000","last_change":"2023-10-03T18:11:03.377802+0000","last_active":"2023-10-03T18:11:03.636292+0000","last_peered":"2023-10-03T18:11:03.636292+0000","last_clean":"2023-10-03T18:11:03.636292+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:11:03.636292+0000","last_undegraded":"2023-10-03T18:11:03.636292+0000","last_fullsized":"2023-10-03T18:11:03.636292+0000","mapping_epoch":12,"log_start":"119'119131","ondisk_log_start":"119'119131","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":9,"snaptrim_duration":0.58017776899999995,"stat_sum":{"num_bytes":2605183599,"num_objects":3883,"num_object_clones":3279,"num_object_copies":7766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3883,"num_whiteouts":0,"num_read":124018,"num_read_kb":462430,"num_write":116757,"num_write_kb":466440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"4"}]},{"pgid":"2.5","version":"152'155638","reported_seq":289677,"reported_epoch":152,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:03.551691+0000","last_change":"2023-10-03T18:11:03.475394+0000","last_active":"2023-10-03T18:11:03.551691+0000","last_peered":"2023-10-03T18:11:03.551691+0000","last_clean":"2023-10-03T18:11:03.551691+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:11:03.551691+0000","last_undegraded":"2023-10-03T18:11:03.551691+0000","last_fullsized":"2023-10-03T18:11:03.551691+0000","mapping_epoch":12,"log_start":"126'145610","ondisk_log_start":"126'145610","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.791481579,"stat_sum":{"num_bytes":3054412482,"num_objects":4965,"num_object_clones":4254,"num_object_copies":9930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4965,"num_whiteouts":0,"num_read":148352,"num_read_kb":543078,"num_write":137183,"num_write_kb":547643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"2"}]},{"pgid":"2.0","version":"151'146369","reported_seq":274404,"reported_epoch":151,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:11:02.563206+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:11:02.563206+0000","last_peered":"2023-10-03T18:11:02.563206+0000","last_clean":"2023-10-03T18:11:02.563206+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:11:02.563206+0000","last_undegraded":"2023-10-03T18:11:02.563206+0000","last_fullsized":"2023-10-03T18:11:02.563206+0000","mapping_epoch":12,"log_start":"119'136305","ondisk_log_start":"119'136305","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1737,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 198s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2951713475,"num_objects":5181,"num_object_clones":4495,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5181,"num_whiteouts":0,"num_read":138181,"num_read_kb":521041,"num_write":132790,"num_write_kb":530552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":308,"reported_epoch":151,"state":"active+clean","last_fresh":"2023-10-03T18:11:01.743195+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:11:01.743195+0000","last_peered":"2023-10-03T18:11:01.743195+0000","last_clean":"2023-10-03T18:11:01.743195+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:11:01.743195+0000","last_undegraded":"2023-10-03T18:11:01.743195+0000","last_fullsized":"2023-10-03T18:11:01.743195+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"152'149667","reported_seq":279993,"reported_epoch":152,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:03.439849+0000","last_change":"2023-10-03T18:11:01.594426+0000","last_active":"2023-10-03T18:11:03.439849+0000","last_peered":"2023-10-03T18:11:03.439849+0000","last_clean":"2023-10-03T18:11:03.439849+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:11:03.439849+0000","last_undegraded":"2023-10-03T18:11:03.439849+0000","last_fullsized":"2023-10-03T18:11:03.439849+0000","mapping_epoch":12,"log_start":"126'139630","ondisk_log_start":"126'139630","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":2.1404526920000002,"stat_sum":{"num_bytes":3001189093,"num_objects":4721,"num_object_clones":4024,"num_object_copies":9442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4721,"num_whiteouts":0,"num_read":141394,"num_read_kb":532140,"num_write":134474,"num_write_kb":537183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"6"}]},{"pgid":"2.1","version":"151'138302","reported_seq":257494,"reported_epoch":151,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:02.582042+0000","last_change":"2023-10-03T18:11:01.350687+0000","last_active":"2023-10-03T18:11:02.582042+0000","last_peered":"2023-10-03T18:11:02.582042+0000","last_clean":"2023-10-03T18:11:02.582042+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:11:02.582042+0000","last_undegraded":"2023-10-03T18:11:02.582042+0000","last_fullsized":"2023-10-03T18:11:02.582042+0000","mapping_epoch":12,"log_start":"126'128204","ondisk_log_start":"126'128204","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:19:58.800803+0000","scrub_duration":424.08306342600002,"objects_trimmed":14,"snaptrim_duration":0.088567172999999999,"stat_sum":{"num_bytes":2734776842,"num_objects":4281,"num_object_clones":3648,"num_object_copies":8562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4281,"num_whiteouts":0,"num_read":130508,"num_read_kb":483064,"num_write":122619,"num_write_kb":489633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"6"}]},{"pgid":"2.2","version":"153'140145","reported_seq":262404,"reported_epoch":153,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:04.078239+0000","last_change":"2023-10-03T18:11:03.656751+0000","last_active":"2023-10-03T18:11:04.078239+0000","last_peered":"2023-10-03T18:11:04.078239+0000","last_clean":"2023-10-03T18:11:04.078239+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:11:04.078239+0000","last_undegraded":"2023-10-03T18:11:04.078239+0000","last_fullsized":"2023-10-03T18:11:04.078239+0000","mapping_epoch":12,"log_start":"126'130048","ondisk_log_start":"126'130048","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:32:31.941957+0000","scrub_duration":370.574046178,"objects_trimmed":2,"snaptrim_duration":0.89474943799999995,"stat_sum":{"num_bytes":2738311991,"num_objects":3397,"num_object_clones":2754,"num_object_copies":6794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3397,"num_whiteouts":0,"num_read":136130,"num_read_kb":490732,"num_write":123778,"num_write_kb":493869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"8"}]},{"pgid":"2.4","version":"152'136606","reported_seq":255913,"reported_epoch":152,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:03.539294+0000","last_change":"2023-10-03T18:11:01.140575+0000","last_active":"2023-10-03T18:11:03.539294+0000","last_peered":"2023-10-03T18:11:03.539294+0000","last_clean":"2023-10-03T18:11:03.539294+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:11:03.539294+0000","last_undegraded":"2023-10-03T18:11:03.539294+0000","last_fullsized":"2023-10-03T18:11:03.539294+0000","mapping_epoch":12,"log_start":"122'126529","ondisk_log_start":"122'126529","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.91801796300000005,"stat_sum":{"num_bytes":2725233378,"num_objects":4119,"num_object_clones":3481,"num_object_copies":8238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4119,"num_whiteouts":0,"num_read":129420,"num_read_kb":482111,"num_write":122104,"num_write_kb":487745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"5d","length":"6"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22631867767,"num_objects":37142,"num_object_clones":31876,"num_object_copies":74284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37142,"num_whiteouts":0,"num_read":1080835,"num_read_kb":4014654,"num_write":1016299,"num_write_kb":4058775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8310419456,"data_stored":53986442068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":48909,"internal_metadata":0},"log_size":80519,"ondisk_log_size":80519,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2108652,"kb_used_data":1558240,"kb_used_omap":969,"kb_used_meta":549430,"kb_avail":92263188,"statfs":{"total":96636764160,"available":94477504512,"internally_reserved":0,"allocated":1595637760,"data_stored":10322114363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":562617198},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2,6,3,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":3.2599999999999998,"5min":2.5390000000000001,"15min":2.5840000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":17.228999999999999,"5min":32.173000000000002,"15min":32.173000000000002},"last":3.782},{"interface":"front","average":{"1min":1.401,"5min":1.9650000000000001,"15min":2.5619999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":4.6829999999999998,"5min":16.285,"15min":18.041},"last":1.1499999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.9119999999999999,"15min":2.5920000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":7.4189999999999996,"5min":15.08,"15min":26.074000000000002},"last":4.8929999999999998},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.544,"15min":2.2429999999999999},"min":{"1min":0.26800000000000002,"5min":0.188,"15min":0.188},"max":{"1min":7.2919999999999998,"5min":19.097000000000001,"15min":19.097000000000001},"last":4.9100000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738828,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4426988,"kb_used_data":2502428,"kb_used_omap":347,"kb_used_meta":1924196,"kb_avail":89944852,"statfs":{"total":96636764160,"available":92103528448,"internally_reserved":0,"allocated":2562486272,"data_stored":16644330157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":356176,"internal_metadata":1970376880},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":4.3700000000000001,"5min":2.875,"15min":3.6360000000000001},"min":{"1min":0.23499999999999999,"5min":0.17799999999999999,"15min":0.17299999999999999},"max":{"1min":13.180999999999999,"5min":15.081,"15min":30.852},"last":2.7090000000000001},{"interface":"front","average":{"1min":2.5129999999999999,"5min":2.1850000000000001,"15min":2.8239999999999998},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":10.225,"5min":12.289999999999999,"15min":18.492000000000001},"last":0.97499999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:10:14 2023","interfaces":[{"interface":"back","average":{"1min":1.839,"5min":1.603,"15min":2.177},"min":{"1min":0.182,"5min":0.182,"15min":0.182},"max":{"1min":12.228999999999999,"5min":12.228999999999999,"15min":14.195},"last":0.442},{"interface":"front","average":{"1min":2.415,"5min":1.9470000000000001,"15min":2.1659999999999999},"min":{"1min":0.216,"5min":0.216,"15min":0.20599999999999999},"max":{"1min":12.880000000000001,"5min":12.880000000000001,"15min":12.880000000000001},"last":1.2929999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738816,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6068728,"kb_used_data":4059184,"kb_used_omap":1918,"kb_used_meta":2007617,"kb_avail":88303112,"statfs":{"total":96636764160,"available":90422386688,"internally_reserved":0,"allocated":4156604416,"data_stored":26939923090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1964794,"internal_metadata":2055800070},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,4,10,14,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.8959999999999999,"5min":2.4329999999999998,"15min":2.9020000000000001},"min":{"1min":0.29299999999999998,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":8.2609999999999992,"5min":13.519,"15min":16.997},"last":6.1740000000000004},{"interface":"front","average":{"1min":1.472,"5min":2.1379999999999999,"15min":3.1850000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.5110000000000001,"5min":10.5,"15min":24.481000000000002},"last":0.219}]},{"osd":2,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":2.298,"15min":2.8410000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":21.326000000000001,"5min":21.326000000000001,"15min":24.579000000000001},"last":5.2850000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":2.044,"15min":2.3900000000000001},"min":{"1min":0.32000000000000001,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":5.6289999999999996,"5min":10.890000000000001,"15min":20.108000000000001},"last":0.32000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2560884736,"data_stored":16676367966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12935,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4155256832,"data_stored":26994439492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28967,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1594277888,"data_stored":10315634610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:11:07.968 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:07.968 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:07.968 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:07.968 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:07.969 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:07.969 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:11:08.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.053+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:08.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.053+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:08.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.093+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:08.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.124+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:08.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.168+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:08.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.168+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:08.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.238+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:08.253 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.252+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:08.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:08.282+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:10.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.018+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:10.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.106+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:10.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.173+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:10.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.173+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:10.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.175+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:10.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.221+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:10.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:10.223+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:11.472 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.468+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:11.537 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.535+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:11.537 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.535+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:11.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.535+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:11.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.701+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:11.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.793+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:11.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.793+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:11.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:11.793+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:12.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.000+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:12.087 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.086+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:12.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.160+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:12.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.172+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:12.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.172+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:12.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.183+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:12.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.224+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:12.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:12.224+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:13.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.192+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:13.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.192+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:13.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.345+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:13.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.364+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:13.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.364+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:13.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.370+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:13.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.457+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:13.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:13.457+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:14.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.787+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:11:14.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.787+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:14.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:14.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:14.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.840+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:14.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.861+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:14.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.898+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:14.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.898+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:14.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:14.913+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:15.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:15.100+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:15.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:15.220+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:18.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.007+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:18.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.007+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:18.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.053+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:18.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.172+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:18.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.172+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:18.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.258+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:18.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.265+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:18.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:18.333+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:20.438 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.438+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:20.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:20.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:20.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:20.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:20.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:20.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:20.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:20.567+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:20.875 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:20.874+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:11:25.868 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:25.868+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:11:26.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.480+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:26.495 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.495+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:26.495 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.495+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:26.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.495+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:26.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.518+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:26.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:26.519+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:27.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:11:28.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.030+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:28.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.186+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:28.208 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.207+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:28.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.217+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:28.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.217+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:28.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.325+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:28.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.325+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:28.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.325+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:28.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.352+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:28.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.440+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:28.634 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:11:28.648 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:11:28.670 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1309,"stamp":"2023-10-03T18:11:26.772087+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22653777713,"num_objects":39262,"num_object_clones":33996,"num_object_copies":78524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39262,"num_whiteouts":0,"num_read":1088124,"num_read_kb":4036006,"num_write":1021683,"num_write_kb":4081355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80456,"ondisk_log_size":80456,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12671872,"kb_used_data":8160032,"kb_used_omap":5685,"kb_used_meta":4506122,"kb_avail":270443648,"statfs":{"total":289910292480,"available":276934295552,"internally_reserved":0,"allocated":8355872768,"data_stored":57430771779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5822418,"internal_metadata":4614268974},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,0,4,5,9,25,16,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11038720,"num_objects":791,"num_object_clones":791,"num_object_copies":1582,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":791,"num_whiteouts":0,"num_read":2569,"num_read_kb":10169,"num_write":2671,"num_write_kb":10684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002526"},"pg_stats":[{"pgid":"2.7","version":"157'141011","reported_seq":264791,"reported_epoch":157,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:25.430107+0000","last_change":"2023-10-03T18:11:23.715745+0000","last_active":"2023-10-03T18:11:25.430107+0000","last_peered":"2023-10-03T18:11:25.430107+0000","last_clean":"2023-10-03T18:11:25.430107+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:11:25.430107+0000","last_undegraded":"2023-10-03T18:11:25.430107+0000","last_fullsized":"2023-10-03T18:11:25.430107+0000","mapping_epoch":12,"log_start":"126'130975","ondisk_log_start":"126'130975","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":131,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:55:52.990222+0000","scrub_duration":505.38444223300002,"objects_trimmed":26,"snaptrim_duration":11.989521106,"stat_sum":{"num_bytes":2823824046,"num_objects":6898,"num_object_clones":6243,"num_object_copies":13796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6898,"num_whiteouts":0,"num_read":133609,"num_read_kb":502704,"num_write":127276,"num_write_kb":508424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"157'130229","reported_seq":244670,"reported_epoch":157,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:25.321562+0000","last_change":"2023-10-03T18:11:25.035691+0000","last_active":"2023-10-03T18:11:25.321562+0000","last_peered":"2023-10-03T18:11:25.321562+0000","last_clean":"2023-10-03T18:11:25.321562+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:11:25.321562+0000","last_undegraded":"2023-10-03T18:11:25.321562+0000","last_fullsized":"2023-10-03T18:11:25.321562+0000","mapping_epoch":12,"log_start":"126'120137","ondisk_log_start":"126'120137","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":13.052983363999999,"stat_sum":{"num_bytes":2607546940,"num_objects":4167,"num_object_clones":3564,"num_object_copies":8334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":124744,"num_read_kb":464840,"num_write":117324,"num_write_kb":468704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"6"}]},{"pgid":"2.5","version":"157'157044","reported_seq":291420,"reported_epoch":157,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:25.244543+0000","last_change":"2023-10-03T18:11:25.022667+0000","last_active":"2023-10-03T18:11:25.244543+0000","last_peered":"2023-10-03T18:11:25.244543+0000","last_clean":"2023-10-03T18:11:25.244543+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:11:25.244543+0000","last_undegraded":"2023-10-03T18:11:25.244543+0000","last_fullsized":"2023-10-03T18:11:25.244543+0000","mapping_epoch":12,"log_start":"126'147020","ondisk_log_start":"126'147020","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":65,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":3.1939716809999998,"stat_sum":{"num_bytes":3057340969,"num_objects":5282,"num_object_clones":4574,"num_object_copies":10564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":149416,"num_read_kb":546179,"num_write":137905,"num_write_kb":550479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"5"}]},{"pgid":"2.0","version":"157'147611","reported_seq":275959,"reported_epoch":157,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:11:22.866221+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:11:22.866221+0000","last_peered":"2023-10-03T18:11:22.866221+0000","last_clean":"2023-10-03T18:11:22.866221+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:11:22.866221+0000","last_undegraded":"2023-10-03T18:11:22.866221+0000","last_fullsized":"2023-10-03T18:11:22.866221+0000","mapping_epoch":12,"log_start":"126'137609","ondisk_log_start":"126'137609","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":1936,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":80,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 218s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2954351299,"num_objects":5591,"num_object_clones":4905,"num_object_copies":11182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5591,"num_whiteouts":0,"num_read":139087,"num_read_kb":523748,"num_write":133442,"num_write_kb":533106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":313,"reported_epoch":156,"state":"active+clean","last_fresh":"2023-10-03T18:11:06.877662+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:11:06.877662+0000","last_peered":"2023-10-03T18:11:06.877662+0000","last_clean":"2023-10-03T18:11:06.877662+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:11:06.877662+0000","last_undegraded":"2023-10-03T18:11:06.877662+0000","last_fullsized":"2023-10-03T18:11:06.877662+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"157'151021","reported_seq":281565,"reported_epoch":157,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:25.186092+0000","last_change":"2023-10-03T18:11:23.670618+0000","last_active":"2023-10-03T18:11:25.186092+0000","last_peered":"2023-10-03T18:11:25.186092+0000","last_clean":"2023-10-03T18:11:25.186092+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:11:25.186092+0000","last_undegraded":"2023-10-03T18:11:25.186092+0000","last_fullsized":"2023-10-03T18:11:25.186092+0000","mapping_epoch":12,"log_start":"126'140934","ondisk_log_start":"126'140934","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":61,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":1.812375399,"stat_sum":{"num_bytes":3004211890,"num_objects":5109,"num_object_clones":4413,"num_object_copies":10218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5109,"num_whiteouts":0,"num_read":142242,"num_read_kb":534855,"num_write":135211,"num_write_kb":540107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"2"}]},{"pgid":"2.1","version":"157'139928","reported_seq":259305,"reported_epoch":157,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:22.862732+0000","last_change":"2023-10-03T18:11:21.420112+0000","last_active":"2023-10-03T18:11:22.862732+0000","last_peered":"2023-10-03T18:11:22.862732+0000","last_clean":"2023-10-03T18:11:22.862732+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:11:22.862732+0000","last_undegraded":"2023-10-03T18:11:22.862732+0000","last_fullsized":"2023-10-03T18:11:22.862732+0000","mapping_epoch":12,"log_start":"133'129917","ondisk_log_start":"133'129917","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:16:07.470962+0000","scrub_duration":424.08306342600002,"objects_trimmed":16,"snaptrim_duration":14.613302086999999,"stat_sum":{"num_bytes":2737246730,"num_objects":4341,"num_object_clones":3708,"num_object_copies":8682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4341,"num_whiteouts":0,"num_read":131543,"num_read_kb":485635,"num_write":123238,"num_write_kb":492033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"7"}]},{"pgid":"2.2","version":"157'141755","reported_seq":264200,"reported_epoch":157,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:24.311865+0000","last_change":"2023-10-03T18:11:23.343495+0000","last_active":"2023-10-03T18:11:24.311865+0000","last_peered":"2023-10-03T18:11:24.311865+0000","last_clean":"2023-10-03T18:11:24.311865+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:11:24.311865+0000","last_undegraded":"2023-10-03T18:11:24.311865+0000","last_fullsized":"2023-10-03T18:11:24.311865+0000","mapping_epoch":12,"log_start":"133'131658","ondisk_log_start":"133'131658","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:47:35.519265+0000","scrub_duration":370.574046178,"objects_trimmed":40,"snaptrim_duration":1.58297493,"stat_sum":{"num_bytes":2740769591,"num_objects":3450,"num_object_clones":2807,"num_object_copies":6900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":137106,"num_read_kb":493223,"num_write":124391,"num_write_kb":496278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"}]},{"pgid":"2.4","version":"157'137843","reported_seq":257427,"reported_epoch":157,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:25.179283+0000","last_change":"2023-10-03T18:11:21.771673+0000","last_active":"2023-10-03T18:11:25.179283+0000","last_peered":"2023-10-03T18:11:25.179283+0000","last_clean":"2023-10-03T18:11:25.179283+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:11:25.179283+0000","last_undegraded":"2023-10-03T18:11:25.179283+0000","last_fullsized":"2023-10-03T18:11:25.179283+0000","mapping_epoch":12,"log_start":"126'127835","ondisk_log_start":"126'127835","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":14.979870445,"stat_sum":{"num_bytes":2727895880,"num_objects":4422,"num_object_clones":3782,"num_object_copies":8844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4422,"num_whiteouts":0,"num_read":130271,"num_read_kb":484731,"num_write":122759,"num_write_kb":490330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"60","length":"3"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22653187345,"num_objects":39260,"num_object_clones":33996,"num_object_copies":78520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39260,"num_whiteouts":0,"num_read":1088018,"num_read_kb":4035915,"num_write":1021546,"num_write_kb":4079461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8352350208,"data_stored":57491790370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":49912,"internal_metadata":0},"log_size":80357,"ondisk_log_size":80357,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2161992,"kb_used_data":1565756,"kb_used_omap":969,"kb_used_meta":595254,"kb_avail":92209848,"statfs":{"total":96636764160,"available":94422884352,"internally_reserved":0,"allocated":1603334144,"data_stored":11009170122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":609540974},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,2,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":3.2599999999999998,"5min":2.5390000000000001,"15min":2.5840000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":17.228999999999999,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.9650000000000001,"15min":2.5619999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":4.6829999999999998,"5min":16.285,"15min":18.041},"last":4.6710000000000003}]},{"osd":1,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.9119999999999999,"15min":2.5920000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":7.4189999999999996,"5min":15.08,"15min":26.074000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.544,"15min":2.2429999999999999},"min":{"1min":0.26800000000000002,"5min":0.188,"15min":0.188},"max":{"1min":7.2919999999999998,"5min":19.097000000000001,"15min":19.097000000000001},"last":0.52300000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738832,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4425104,"kb_used_data":2514312,"kb_used_omap":2413,"kb_used_meta":1908370,"kb_avail":89946736,"statfs":{"total":96636764160,"available":92105457664,"internally_reserved":0,"allocated":2574655488,"data_stored":17679344381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2471542,"internal_metadata":1954171274},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,0,3,1,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":3.129,"5min":3.113,"15min":3.5870000000000002},"min":{"1min":0.19600000000000001,"5min":0.17899999999999999,"15min":0.17299999999999999},"max":{"1min":10.895,"5min":15.081,"15min":30.852},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.194,"5min":2.3109999999999999,"15min":2.7200000000000002},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":11.574,"5min":12.289999999999999,"15min":18.492000000000001},"last":0.28699999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.6359999999999999,"15min":2.077},"min":{"1min":0.22,"5min":0.182,"15min":0.182},"max":{"1min":4.5960000000000001,"5min":12.228999999999999,"15min":14.195},"last":0.46200000000000002},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.887,"15min":2.1019999999999999},"min":{"1min":0.224,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":4.593,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.44800000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6084776,"kb_used_data":4079964,"kb_used_omap":2303,"kb_used_meta":2002496,"kb_avail":88287064,"statfs":{"total":96636764160,"available":90405953536,"internally_reserved":0,"allocated":4177883136,"data_stored":28742257276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2358474,"internal_metadata":2050556726},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,3,2,6,16,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.8959999999999999,"5min":2.4329999999999998,"15min":2.9020000000000001},"min":{"1min":0.29299999999999998,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":8.2609999999999992,"5min":13.519,"15min":16.997},"last":7.5910000000000002},{"interface":"front","average":{"1min":1.472,"5min":2.1379999999999999,"15min":3.1850000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.5110000000000001,"5min":10.5,"15min":24.481000000000002},"last":0.72799999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":2.298,"15min":2.8410000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":21.326000000000001,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.88},{"interface":"front","average":{"1min":1.5920000000000001,"5min":2.044,"15min":2.3900000000000001},"min":{"1min":0.32000000000000001,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":5.6289999999999996,"5min":10.890000000000001,"15min":20.108000000000001},"last":2.0209999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2573221888,"data_stored":17704209810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":16032,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4177158144,"data_stored":28779132177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26873,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1601970176,"data_stored":11008448383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:11:28.672 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:28.672 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:28.672 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:28.673 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:28.673 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:28.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:11:28.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:28.997+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:29.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:29.083+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:29.084 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:29.083+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:29.085 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:29.084+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:29.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:29.103+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:29.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:29.228+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:32.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.857+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:32.925 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.925+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:32.953 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.949+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:32.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.959+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:32.976 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.976+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:32.977 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.977+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:32.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.993+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:32.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:32.993+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:33.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:33.025+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:34.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:34.170+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:34.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:34.184+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:34.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:34.184+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:34.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:34.325+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:34.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:34.378+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:35.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.094+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:35.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.101+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:35.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.107+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:35.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.129+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:35.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.129+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:35.203 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.203+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:35.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.203+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:35.217 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.216+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:35.218 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.216+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:35.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:35.244+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:36.970 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:36.963+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:36.970 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:36.964+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:37.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.099+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:37.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.151+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:37.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.151+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:37.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.151+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:37.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.175+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:37.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.175+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:37.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:37.202+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:39.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.515+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:39.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.515+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:39.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.606+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:39.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.606+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:39.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.753+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:39.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.753+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:39.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.753+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:39.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.753+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:39.868 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:39.867+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:40.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.417+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:40.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.417+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:40.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.447+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:40.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.447+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:40.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.447+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:40.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.447+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:40.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.447+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:40.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.639+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:40.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.717+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:40.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.717+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:40.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.738+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:40.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.920+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:40.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.920+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:40.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.920+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:40.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.920+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:40.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.920+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:40.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.932+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:11:40.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.953+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:40.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:40.953+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:11:41.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.001+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:41.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.028+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:41.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.038+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:41.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.071+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:41.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.071+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:41.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.071+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:41.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.114+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:41.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:41.175+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:44.517 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.516+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:11:44.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.518+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:44.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.573+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:44.575 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.574+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:44.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.628+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:44.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.628+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:44.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.628+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:44.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.661+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:44.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:44.708+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:45.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.698+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:45.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.698+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:45.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.711+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:45.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.711+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:45.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.711+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:45.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.712+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:45.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:45.760+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:48.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:11:48.895 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:48.884+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:11:48.896 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:48.884+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:11:48.896 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:48.884+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:11:48.896 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:11:48.884+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:11:49.462 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:11:49.465 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:11:49.486 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1323,"stamp":"2023-10-03T18:11:48.795761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22681474865,"num_objects":39240,"num_object_clones":33974,"num_object_copies":78480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39240,"num_whiteouts":0,"num_read":1094618,"num_read_kb":4061918,"num_write":1028329,"num_write_kb":4107939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80468,"ondisk_log_size":80468,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":10834176,"kb_used_data":8212832,"kb_used_omap":5694,"kb_used_meta":2615617,"kb_avail":272281344,"statfs":{"total":289910292480,"available":278816096256,"internally_reserved":0,"allocated":8409939968,"data_stored":57641195683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":2678392781},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,5,12,24,15,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12771328,"num_objects":-181,"num_object_clones":-181,"num_object_copies":-362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-181,"num_whiteouts":0,"num_read":3057,"num_read_kb":12196,"num_write":3057,"num_write_kb":12228,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003760"},"pg_stats":[{"pgid":"2.7","version":"160'142196","reported_seq":266577,"reported_epoch":160,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:47.099581+0000","last_change":"2023-10-03T18:11:46.412725+0000","last_active":"2023-10-03T18:11:47.099581+0000","last_peered":"2023-10-03T18:11:47.099581+0000","last_clean":"2023-10-03T18:11:47.099581+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:11:47.099581+0000","last_undegraded":"2023-10-03T18:11:47.099581+0000","last_fullsized":"2023-10-03T18:11:47.099581+0000","mapping_epoch":12,"log_start":"126'132179","ondisk_log_start":"126'132179","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":127,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:55:52.990222+0000","scrub_duration":505.38444223300002,"objects_trimmed":4,"snaptrim_duration":1.6177253679999999,"stat_sum":{"num_bytes":2827092654,"num_objects":6932,"num_object_clones":6277,"num_object_copies":13864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6932,"num_whiteouts":0,"num_read":134388,"num_read_kb":505816,"num_write":128052,"num_write_kb":511528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"66","length":"1"}]},{"pgid":"2.6","version":"160'131398","reported_seq":246342,"reported_epoch":160,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:47.093961+0000","last_change":"2023-10-03T18:11:46.412616+0000","last_active":"2023-10-03T18:11:47.093961+0000","last_peered":"2023-10-03T18:11:47.093961+0000","last_clean":"2023-10-03T18:11:47.093961+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:11:47.093961+0000","last_undegraded":"2023-10-03T18:11:47.093961+0000","last_fullsized":"2023-10-03T18:11:47.093961+0000","mapping_epoch":12,"log_start":"126'121341","ondisk_log_start":"126'121341","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.6225632780000003,"stat_sum":{"num_bytes":2610659900,"num_objects":4232,"num_object_clones":3629,"num_object_copies":8464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4232,"num_whiteouts":0,"num_read":125449,"num_read_kb":467648,"num_write":118073,"num_write_kb":471700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"1"}]},{"pgid":"2.5","version":"160'158364","reported_seq":293396,"reported_epoch":160,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:47.092027+0000","last_change":"2023-10-03T18:11:45.902791+0000","last_active":"2023-10-03T18:11:47.092027+0000","last_peered":"2023-10-03T18:11:47.092027+0000","last_clean":"2023-10-03T18:11:47.092027+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:11:47.092027+0000","last_undegraded":"2023-10-03T18:11:47.092027+0000","last_fullsized":"2023-10-03T18:11:47.092027+0000","mapping_epoch":12,"log_start":"133'148323","ondisk_log_start":"133'148323","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.1211062679999999,"stat_sum":{"num_bytes":3061051945,"num_objects":5360,"num_object_clones":4652,"num_object_copies":10720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5360,"num_whiteouts":0,"num_read":150261,"num_read_kb":549543,"num_write":138801,"num_write_kb":554063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"1"}]},{"pgid":"2.0","version":"160'148993","reported_seq":278125,"reported_epoch":160,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:11:47.934246+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:11:47.934246+0000","last_peered":"2023-10-03T18:11:47.934246+0000","last_clean":"2023-10-03T18:11:47.934246+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:11:47.934246+0000","last_undegraded":"2023-10-03T18:11:47.934246+0000","last_fullsized":"2023-10-03T18:11:47.934246+0000","mapping_epoch":12,"log_start":"126'138912","ondisk_log_start":"126'138912","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2180,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":80,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 243s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2958348995,"num_objects":5785,"num_object_clones":5099,"num_object_copies":11570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5785,"num_whiteouts":0,"num_read":140034,"num_read_kb":527524,"num_write":134413,"num_write_kb":536990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":317,"reported_epoch":159,"state":"active+clean","last_fresh":"2023-10-03T18:11:41.865415+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:11:41.865415+0000","last_peered":"2023-10-03T18:11:41.865415+0000","last_clean":"2023-10-03T18:11:41.865415+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:11:41.865415+0000","last_undegraded":"2023-10-03T18:11:41.865415+0000","last_fullsized":"2023-10-03T18:11:41.865415+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"160'152316","reported_seq":283580,"reported_epoch":160,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:47.045443+0000","last_change":"2023-10-03T18:11:45.900658+0000","last_active":"2023-10-03T18:11:47.045443+0000","last_peered":"2023-10-03T18:11:47.045443+0000","last_clean":"2023-10-03T18:11:47.045443+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:11:47.045443+0000","last_undegraded":"2023-10-03T18:11:47.045443+0000","last_fullsized":"2023-10-03T18:11:47.045443+0000","mapping_epoch":12,"log_start":"127'142239","ondisk_log_start":"127'142239","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":57,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.1126885609999997,"stat_sum":{"num_bytes":3007832754,"num_objects":5134,"num_object_clones":4438,"num_object_copies":10268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5134,"num_whiteouts":0,"num_read":143143,"num_read_kb":538451,"num_write":136083,"num_write_kb":543595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"3"}]},{"pgid":"2.1","version":"160'141868","reported_seq":261809,"reported_epoch":160,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:47.913897+0000","last_change":"2023-10-03T18:11:46.978546+0000","last_active":"2023-10-03T18:11:47.913897+0000","last_peered":"2023-10-03T18:11:47.913897+0000","last_clean":"2023-10-03T18:11:47.913897+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:11:47.913897+0000","last_undegraded":"2023-10-03T18:11:47.913897+0000","last_fullsized":"2023-10-03T18:11:47.913897+0000","mapping_epoch":12,"log_start":"133'131825","ondisk_log_start":"133'131825","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:16:07.470962+0000","scrub_duration":424.08306342600002,"objects_trimmed":25,"snaptrim_duration":0.17991480900000001,"stat_sum":{"num_bytes":2740986378,"num_objects":4069,"num_object_clones":3436,"num_object_copies":8138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4069,"num_whiteouts":0,"num_read":132415,"num_read_kb":489123,"num_write":124124,"num_write_kb":495577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"2"}]},{"pgid":"2.2","version":"159'143317","reported_seq":266216,"reported_epoch":159,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:11:44.522154+0000","last_change":"2023-10-03T18:11:43.684108+0000","last_active":"2023-10-03T18:11:44.522154+0000","last_peered":"2023-10-03T18:11:44.522154+0000","last_clean":"2023-10-03T18:11:44.522154+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:11:44.522154+0000","last_undegraded":"2023-10-03T18:11:44.522154+0000","last_fullsized":"2023-10-03T18:11:44.522154+0000","mapping_epoch":12,"log_start":"133'133267","ondisk_log_start":"133'133267","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:47:35.519265+0000","scrub_duration":370.574046178,"objects_trimmed":54,"snaptrim_duration":1.9032451969999999,"stat_sum":{"num_bytes":2743747383,"num_objects":3268,"num_object_clones":2625,"num_object_copies":6536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3268,"num_whiteouts":0,"num_read":137820,"num_read_kb":496075,"num_write":125106,"num_write_kb":499138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"62","length":"1"},{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"}]},{"pgid":"2.4","version":"160'139043","reported_seq":259171,"reported_epoch":160,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:11:47.077314+0000","last_change":"2023-10-03T18:11:44.715780+0000","last_active":"2023-10-03T18:11:47.077314+0000","last_peered":"2023-10-03T18:11:47.077314+0000","last_clean":"2023-10-03T18:11:47.077314+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:11:47.077314+0000","last_undegraded":"2023-10-03T18:11:47.077314+0000","last_fullsized":"2023-10-03T18:11:47.077314+0000","mapping_epoch":12,"log_start":"126'129040","ondisk_log_start":"126'129040","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":51,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.9298297619999998,"stat_sum":{"num_bytes":2731164488,"num_objects":4458,"num_object_clones":3818,"num_object_copies":8916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4458,"num_whiteouts":0,"num_read":131002,"num_read_kb":487647,"num_write":123540,"num_write_kb":493454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"66","length":"12"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"9"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22680884497,"num_objects":39238,"num_object_clones":33974,"num_object_copies":78476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39238,"num_whiteouts":0,"num_read":1094512,"num_read_kb":4061827,"num_write":1028192,"num_write_kb":4106045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8405270528,"data_stored":57620306466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80369,"ondisk_log_size":80369,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2240100,"kb_used_data":1577244,"kb_used_omap":969,"kb_used_meta":661878,"kb_avail":92131740,"statfs":{"total":96636764160,"available":94342901760,"internally_reserved":0,"allocated":1615097856,"data_stored":11041972458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":677763950},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":3.2599999999999998,"5min":2.5390000000000001,"15min":2.5840000000000001},"min":{"1min":0.16600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":17.228999999999999,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.19600000000000001},{"interface":"front","average":{"1min":1.401,"5min":1.9650000000000001,"15min":2.5619999999999998},"min":{"1min":0.24399999999999999,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":4.6829999999999998,"5min":16.285,"15min":18.041},"last":6.657}]},{"osd":1,"last update":"Tue Oct 3 18:11:00 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":2.9119999999999999,"15min":2.5920000000000001},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":7.4189999999999996,"5min":15.08,"15min":26.074000000000002},"last":11.539},{"interface":"front","average":{"1min":2.5910000000000002,"5min":2.544,"15min":2.2429999999999999},"min":{"1min":0.26800000000000002,"5min":0.188,"15min":0.188},"max":{"1min":7.2919999999999998,"5min":19.097000000000001,"15min":19.097000000000001},"last":2.1379999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738836,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3461012,"kb_used_data":2528648,"kb_used_omap":2419,"kb_used_meta":929932,"kb_avail":90910828,"statfs":{"total":96636764160,"available":93092687872,"internally_reserved":0,"allocated":2589335552,"data_stored":17793824541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":952250640},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,1,5,4,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":3.129,"5min":3.113,"15min":3.5870000000000002},"min":{"1min":0.19600000000000001,"5min":0.17899999999999999,"15min":0.17299999999999999},"max":{"1min":10.895,"5min":15.081,"15min":30.852},"last":25.242000000000001},{"interface":"front","average":{"1min":2.194,"5min":2.3109999999999999,"15min":2.7200000000000002},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":11.574,"5min":12.289999999999999,"15min":18.492000000000001},"last":8.9920000000000009}]},{"osd":2,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.6359999999999999,"15min":2.077},"min":{"1min":0.22,"5min":0.182,"15min":0.182},"max":{"1min":4.5960000000000001,"5min":12.228999999999999,"15min":14.195},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.887,"15min":2.1019999999999999},"min":{"1min":0.224,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":4.593,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.247}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5133064,"kb_used_data":4106940,"kb_used_omap":2305,"kb_used_meta":1023806,"kb_avail":89238776,"statfs":{"total":96636764160,"available":91380506624,"internally_reserved":0,"allocated":4205506560,"data_stored":28805398684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1048378191},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,4,11,17,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.8959999999999999,"5min":2.4329999999999998,"15min":2.9020000000000001},"min":{"1min":0.29299999999999998,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":8.2609999999999992,"5min":13.519,"15min":16.997},"last":6.8869999999999996},{"interface":"front","average":{"1min":1.472,"5min":2.1379999999999999,"15min":3.1850000000000001},"min":{"1min":0.219,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":5.5110000000000001,"5min":10.5,"15min":24.481000000000002},"last":0.249}]},{"osd":2,"last update":"Tue Oct 3 18:11:02 2023","interfaces":[{"interface":"back","average":{"1min":2.7469999999999999,"5min":2.298,"15min":2.8410000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":21.326000000000001,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.5920000000000001,"5min":2.044,"15min":2.3900000000000001},"min":{"1min":0.32000000000000001,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":5.6289999999999996,"5min":10.890000000000001,"15min":20.108000000000001},"last":0.29699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2587947008,"data_stored":17781857682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4203614208,"data_stored":28803441937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1613709312,"data_stored":11035006847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:11:49.488 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:49.489 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:49.489 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:49.489 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:49.489 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:11:49.490 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:11:51.426 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:11:51.424+0000 7f5801100700 -1 calc_pg_upmaps abort due to max <= 0 2023-10-03T18:11:55.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.441+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:55.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.476+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:55.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.476+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:55.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.477+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:55.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.477+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:55.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.477+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:55.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.491+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:55.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:55.491+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:57.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.781+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:57.845 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.844+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:57.845 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.844+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:57.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.883+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:57.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.883+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:11:57.952 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:57.951+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:11:58.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.514+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:11:58.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.514+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:11:58.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.514+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:11:58.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.566+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:11:58.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.566+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:11:58.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.566+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:11:58.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:11:58.566+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:00.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.874+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:00.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.874+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:00.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.874+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:00.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.874+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:00.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.882+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:00.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:00.884+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:02.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:02.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:02.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.016+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:02.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.102+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:02.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.234+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:02.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.247+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:02.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.283+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:02.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.283+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:02.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.283+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:02.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.311+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:02.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.313+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:02.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.383+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:02.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.383+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:02.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.383+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:02.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.397+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:02.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.486+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:02.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.533+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:02.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.547+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:02.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.547+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:02.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.549+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:02.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.582+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:02.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.598+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:02.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.606+0000 7fc515f98700 -1 reset not still connected to 0x56220470c340 2023-10-03T18:12:02.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.626+0000 7fc515f98700 -1 reset not still connected to 0x562212e58750 2023-10-03T18:12:02.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.626+0000 7fc515f98700 -1 reset not still connected to 0x562213b8f380 2023-10-03T18:12:02.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.626+0000 7fc515f98700 -1 reset not still connected to 0x562233adc000 2023-10-03T18:12:02.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:02.626+0000 7fc515f98700 -1 reset not still connected to 0x5622377232b0 2023-10-03T18:12:03.619 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:03.618+0000 7fc515f98700 -1 reset not still connected to 0x56220470c340 2023-10-03T18:12:03.815 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:03.814+0000 7fc515f98700 -1 reset not still connected to 0x56220557e680 2023-10-03T18:12:03.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:03.979+0000 7fc515f98700 -1 reset not still connected to 0x5622260a0c30 2023-10-03T18:12:04.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:04.038+0000 7fc515f98700 -1 reset not still connected to 0x562230095380 2023-10-03T18:12:04.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:04.039+0000 7fc515f98700 -1 reset not still connected to 0x562233adc000 2023-10-03T18:12:06.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:06.879+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:06.978 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:06.977+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:07.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.061+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:07.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.061+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:07.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.061+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:07.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.061+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:07.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.135+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:07.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.188+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:07.823 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.821+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:07.823 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.821+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:07.823 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.821+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:07.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.831+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:07.967 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:07.964+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:08.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.045+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:08.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.149+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:08.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.149+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:08.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.166+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:08.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.194+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:08.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:08.223+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:09.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:12:09.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:09.664+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:09.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:09.664+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:09.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:09.718+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:09.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:09.871+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:09.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:09.871+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:10.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.101+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:10.102 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.101+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:10.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.186+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:10.211 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:12:10.240 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:12:10.249 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1343,"stamp":"2023-10-03T18:12:08.801782+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22707124017,"num_objects":39369,"num_object_clones":34103,"num_object_copies":78738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39369,"num_whiteouts":0,"num_read":1104019,"num_read_kb":4089807,"num_write":1034722,"num_write_kb":4132791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80569,"ondisk_log_size":80569,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11023176,"kb_used_data":8262944,"kb_used_omap":5694,"kb_used_meta":2754497,"kb_avail":272092344,"statfs":{"total":289910292480,"available":278622560256,"internally_reserved":0,"allocated":8461254656,"data_stored":57697134032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":2820605901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1,7,7,8,21,12,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5123943,"num_objects":681,"num_object_clones":684,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":681,"num_whiteouts":0,"num_read":2575,"num_read_kb":6163,"num_write":1358,"num_write_kb":5062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.007945"},"pg_stats":[{"pgid":"2.7","version":"170'143452","reported_seq":268514,"reported_epoch":170,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:08.770581+0000","last_change":"2023-10-03T18:12:08.353110+0000","last_active":"2023-10-03T18:12:08.770581+0000","last_peered":"2023-10-03T18:12:08.770581+0000","last_clean":"2023-10-03T18:12:08.770581+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:12:08.770581+0000","last_undegraded":"2023-10-03T18:12:08.770581+0000","last_fullsized":"2023-10-03T18:12:08.770581+0000","mapping_epoch":12,"log_start":"133'133380","ondisk_log_start":"133'133380","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":143,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:37:09.662642+0000","scrub_duration":505.38444223300002,"objects_trimmed":8,"snaptrim_duration":0.845587859,"stat_sum":{"num_bytes":2830336485,"num_objects":7008,"num_object_clones":6357,"num_object_copies":14016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7008,"num_whiteouts":0,"num_read":135490,"num_read_kb":509284,"num_write":128866,"num_write_kb":514696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"170'132542","reported_seq":248146,"reported_epoch":170,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:08.683135+0000","last_change":"2023-10-03T18:12:05.368761+0000","last_active":"2023-10-03T18:12:08.683135+0000","last_peered":"2023-10-03T18:12:08.683135+0000","last_clean":"2023-10-03T18:12:08.683135+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:12:08.683135+0000","last_undegraded":"2023-10-03T18:12:08.683135+0000","last_fullsized":"2023-10-03T18:12:08.683135+0000","mapping_epoch":12,"log_start":"126'122445","ondisk_log_start":"126'122445","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":65,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.50838755099999999,"stat_sum":{"num_bytes":2613613116,"num_objects":4266,"num_object_clones":3663,"num_object_copies":8532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4266,"num_whiteouts":0,"num_read":126404,"num_read_kb":470744,"num_write":118789,"num_write_kb":474528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"5"}]},{"pgid":"2.5","version":"170'159732","reported_seq":295617,"reported_epoch":170,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:08.654373+0000","last_change":"2023-10-03T18:12:06.880139+0000","last_active":"2023-10-03T18:12:08.654373+0000","last_peered":"2023-10-03T18:12:08.654373+0000","last_clean":"2023-10-03T18:12:08.654373+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:12:08.654373+0000","last_undegraded":"2023-10-03T18:12:08.654373+0000","last_fullsized":"2023-10-03T18:12:08.654373+0000","mapping_epoch":12,"log_start":"133'149726","ondisk_log_start":"133'149726","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":78,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033211202000000002,"stat_sum":{"num_bytes":3064648284,"num_objects":5430,"num_object_clones":4721,"num_object_copies":10860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5430,"num_whiteouts":0,"num_read":151751,"num_read_kb":553663,"num_write":139711,"num_write_kb":557570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"2"}]},{"pgid":"2.0","version":"170'150098","reported_seq":279965,"reported_epoch":170,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:12:08.057245+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:12:08.057245+0000","last_peered":"2023-10-03T18:12:08.057245+0000","last_clean":"2023-10-03T18:12:08.057245+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:12:08.057245+0000","last_undegraded":"2023-10-03T18:12:08.057245+0000","last_fullsized":"2023-10-03T18:12:08.057245+0000","mapping_epoch":12,"log_start":"133'140014","ondisk_log_start":"133'140014","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2333,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 263s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2961490676,"num_objects":5965,"num_object_clones":5278,"num_object_copies":11930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5965,"num_whiteouts":0,"num_read":141130,"num_read_kb":530972,"num_write":135197,"num_write_kb":540051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":327,"reported_epoch":169,"state":"active+clean","last_fresh":"2023-10-03T18:12:05.944669+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:12:05.944669+0000","last_peered":"2023-10-03T18:12:05.944669+0000","last_clean":"2023-10-03T18:12:05.944669+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:12:05.944669+0000","last_undegraded":"2023-10-03T18:12:05.944669+0000","last_fullsized":"2023-10-03T18:12:05.944669+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"170'153635","reported_seq":285585,"reported_epoch":170,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:08.652752+0000","last_change":"2023-10-03T18:12:06.978538+0000","last_active":"2023-10-03T18:12:08.652752+0000","last_peered":"2023-10-03T18:12:08.652752+0000","last_clean":"2023-10-03T18:12:08.652752+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:12:08.652752+0000","last_undegraded":"2023-10-03T18:12:08.652752+0000","last_fullsized":"2023-10-03T18:12:08.652752+0000","mapping_epoch":12,"log_start":"133'143544","ondisk_log_start":"133'143544","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.35120015300000001,"stat_sum":{"num_bytes":3011371900,"num_objects":5240,"num_object_clones":4540,"num_object_copies":10480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5240,"num_whiteouts":0,"num_read":144274,"num_read_kb":542050,"num_write":136956,"num_write_kb":547017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"6"}]},{"pgid":"2.1","version":"170'143438","reported_seq":263964,"reported_epoch":170,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:08.026210+0000","last_change":"2023-10-03T18:12:07.894860+0000","last_active":"2023-10-03T18:12:08.026210+0000","last_peered":"2023-10-03T18:12:08.026210+0000","last_clean":"2023-10-03T18:12:08.026210+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:12:08.026210+0000","last_undegraded":"2023-10-03T18:12:08.026210+0000","last_fullsized":"2023-10-03T18:12:08.026210+0000","mapping_epoch":12,"log_start":"134'133436","ondisk_log_start":"134'133436","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:10:15.175002+0000","scrub_duration":424.08306342600002,"objects_trimmed":12,"snaptrim_duration":1.0163987240000001,"stat_sum":{"num_bytes":2743865866,"num_objects":3932,"num_object_clones":3299,"num_object_copies":7864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3932,"num_whiteouts":0,"num_read":133593,"num_read_kb":492469,"num_write":124844,"num_write_kb":498363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"1"}]},{"pgid":"2.2","version":"167'144920","reported_seq":268577,"reported_epoch":167,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:04.572054+0000","last_change":"2023-10-03T18:12:03.853553+0000","last_active":"2023-10-03T18:12:04.572054+0000","last_peered":"2023-10-03T18:12:04.572054+0000","last_clean":"2023-10-03T18:12:04.572054+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:12:04.572054+0000","last_undegraded":"2023-10-03T18:12:04.572054+0000","last_fullsized":"2023-10-03T18:12:04.572054+0000","mapping_epoch":12,"log_start":"133'134876","ondisk_log_start":"133'134876","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:41:12.433306+0000","scrub_duration":370.574046178,"objects_trimmed":16,"snaptrim_duration":0.035579636999999997,"stat_sum":{"num_bytes":2746852151,"num_objects":2996,"num_object_clones":2353,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":139191,"num_read_kb":499620,"num_write":125888,"num_write_kb":502129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"69","length":"f"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"2"}]},{"pgid":"2.4","version":"170'140215","reported_seq":261035,"reported_epoch":170,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:08.683078+0000","last_change":"2023-10-03T18:12:08.353025+0000","last_active":"2023-10-03T18:12:08.683078+0000","last_peered":"2023-10-03T18:12:08.683078+0000","last_clean":"2023-10-03T18:12:08.683078+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:12:08.683078+0000","last_undegraded":"2023-10-03T18:12:08.683078+0000","last_fullsized":"2023-10-03T18:12:08.683078+0000","mapping_epoch":12,"log_start":"132'130141","ondisk_log_start":"132'130141","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":67,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":1.472307928,"stat_sum":{"num_bytes":2734355171,"num_objects":4530,"num_object_clones":3892,"num_object_copies":9060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4530,"num_whiteouts":0,"num_read":132080,"num_read_kb":490914,"num_write":124334,"num_write_kb":496543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"d"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22706533649,"num_objects":39367,"num_object_clones":34103,"num_object_copies":78734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39367,"num_whiteouts":0,"num_read":1103913,"num_read_kb":4089716,"num_write":1034585,"num_write_kb":4130897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8455995392,"data_stored":57782733397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80470,"ondisk_log_size":80470,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738842,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2258936,"kb_used_data":1586348,"kb_used_omap":969,"kb_used_meta":671606,"kb_avail":92112904,"statfs":{"total":96636764160,"available":94323613696,"internally_reserved":0,"allocated":1624420352,"data_stored":11148866605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":687725422},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,0,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":2.694,"15min":2.5329999999999999},"min":{"1min":0.19600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":9.2159999999999993,"5min":32.173000000000002,"15min":32.173000000000002},"last":5.4109999999999996},{"interface":"front","average":{"1min":2.081,"5min":2.0070000000000001,"15min":2.4910000000000001},"min":{"1min":0.22,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":7.556,"5min":16.285,"15min":18.041},"last":0.44400000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.71,"15min":2.5150000000000001},"min":{"1min":0.26100000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":11.539,"5min":15.08,"15min":26.074000000000002},"last":0.82799999999999996},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.6179999999999999,"15min":2.2509999999999999},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":12.853,"5min":14.917,"15min":19.097000000000001},"last":5.6280000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738840,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3506488,"kb_used_data":2544936,"kb_used_omap":2419,"kb_used_meta":959116,"kb_avail":90865352,"statfs":{"total":96636764160,"available":93046120448,"internally_reserved":0,"allocated":2606014464,"data_stored":17581043391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":982135056},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":3.129,"5min":3.113,"15min":3.5870000000000002},"min":{"1min":0.19600000000000001,"5min":0.17899999999999999,"15min":0.17299999999999999},"max":{"1min":10.895,"5min":15.081,"15min":30.852},"last":2.4849999999999999},{"interface":"front","average":{"1min":2.194,"5min":2.3109999999999999,"15min":2.7200000000000002},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":11.574,"5min":12.289999999999999,"15min":18.492000000000001},"last":2.5179999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:11:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.6359999999999999,"15min":2.077},"min":{"1min":0.22,"5min":0.182,"15min":0.182},"max":{"1min":4.5960000000000001,"5min":12.228999999999999,"15min":14.195},"last":0.23300000000000001},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.887,"15min":2.1019999999999999},"min":{"1min":0.224,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":4.593,"5min":12.880000000000001,"15min":12.880000000000001},"last":3.3039999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5257752,"kb_used_data":4131660,"kb_used_omap":2305,"kb_used_meta":1123774,"kb_avail":89114088,"statfs":{"total":96636764160,"available":91252826112,"internally_reserved":0,"allocated":4230819840,"data_stored":28967224036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1150745423},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,7,8,13,11,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":2.5,"15min":2.863},"min":{"1min":0.20599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":9.6400000000000006,"5min":13.519,"15min":16.997},"last":9.6400000000000006},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.99,"15min":3.161},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":8.7919999999999998,"5min":10.5,"15min":24.481000000000002},"last":8.7919999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.1400000000000001,"15min":2.9100000000000001},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.419,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.9550000000000001,"15min":2.2949999999999999},"min":{"1min":0.248,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":7.9020000000000001,"5min":10.890000000000001,"15min":20.108000000000001},"last":2.5390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2603823104,"data_stored":17581554890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4229222400,"data_stored":29034288401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1622949888,"data_stored":11166890106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:12:10.251 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:10.251 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:10.251 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:10.251 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:10.252 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:10.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:12:10.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.368+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:10.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.383+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:10.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.462+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:10.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.462+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:10.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.522+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:10.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.573+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:10.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.603+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:10.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.603+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:10.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.619+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:10.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.619+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:10.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:10.720+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:10.926 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:12:10.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:12:11.568 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:12:11.587 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:12:12.270 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:12:12.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:12:12.985 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:12:13.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:12:13.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.030+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:13.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.072+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:13.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.072+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:13.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.145+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:13.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.146+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:13.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.146+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:13.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.167+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:13.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:13.243+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:13.663 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:12:13.680 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:12:14.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.654+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:14.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.661+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:14.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.810+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:14.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.822+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:14.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.822+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:14.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.822+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:14.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.822+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:14.828 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.827+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:14.828 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:14.827+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:15.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.678+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:15.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.678+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:15.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.678+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:15.691 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.691+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:15.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.695+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:15.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.695+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:15.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.714+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:15.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:15.714+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:18.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:18.162+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:18.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:18.203+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:18.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:18.259+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:18.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:18.259+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:18.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:18.261+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:22.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.155+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:22.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.258+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:22.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.258+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:22.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.592+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:22.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.635+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:22.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:22.635+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:23.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:23.887+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:23.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:23.915+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:24.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.090+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:24.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.091+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:24.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.092+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:24.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.211+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:24.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.211+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:24.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.262+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:24.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.292+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:24.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.295+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:24.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.350+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:24.392 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.390+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:24.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.390+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:24.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.435+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:24.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.435+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:24.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.454+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:24.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.454+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:24.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.454+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:24.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.510+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:24.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.510+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:24.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.510+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:24.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.518+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:24.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.518+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:24.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.518+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:24.613 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.612+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.656+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:24.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.727+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:24.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.748+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:24.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.789+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:24.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:24.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:24.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:24.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:24.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:24.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.880+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:24.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.913+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:24.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.913+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:24.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.943+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:24.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.943+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:24.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.954+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:24.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.996+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:24.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:24.996+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:25.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.023+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:25.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.040+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.162+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:25.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.824+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:25.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.878+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:25.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.878+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:25.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.878+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:25.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.878+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:25.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.885+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:25.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.885+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:25.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.885+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:25.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.885+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:25.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:25.885+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:29.406 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:29.399+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:29.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:29.399+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:29.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:29.399+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:29.423 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:29.423+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:29.430 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:29.430+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:33.681 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:12:34.431 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:12:34.432 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:12:34.449 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1358,"stamp":"2023-10-03T18:12:32.812929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22730307377,"num_objects":40767,"num_object_clones":35501,"num_object_copies":81534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40767,"num_whiteouts":0,"num_read":1111175,"num_read_kb":4113676,"num_write":1040196,"num_write_kb":4154687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80527,"ondisk_log_size":80527,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11396824,"kb_used_data":8307324,"kb_used_omap":5694,"kb_used_meta":3083777,"kb_avail":271718696,"statfs":{"total":289910292480,"available":278239944704,"internally_reserved":0,"allocated":8506699776,"data_stored":60409184379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3157788621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,5,2,5,9,10,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11886592,"num_objects":402,"num_object_clones":402,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":3078,"num_read_kb":11783,"num_write":2820,"num_write_kb":11280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001761"},"pg_stats":[{"pgid":"2.7","version":"172'144629","reported_seq":270150,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:30.675029+0000","last_change":"2023-10-03T18:12:29.274226+0000","last_active":"2023-10-03T18:12:30.675029+0000","last_peered":"2023-10-03T18:12:30.675029+0000","last_clean":"2023-10-03T18:12:30.675029+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:12:30.675029+0000","last_undegraded":"2023-10-03T18:12:30.675029+0000","last_fullsized":"2023-10-03T18:12:30.675029+0000","mapping_epoch":12,"log_start":"133'134582","ondisk_log_start":"133'134582","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:37:09.662642+0000","scrub_duration":505.38444223300002,"objects_trimmed":42,"snaptrim_duration":2.1868002720000002,"stat_sum":{"num_bytes":2833183205,"num_objects":7238,"num_object_clones":6587,"num_object_copies":14476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7238,"num_whiteouts":0,"num_read":136396,"num_read_kb":512464,"num_write":129527,"num_write_kb":517340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"2.6","version":"172'133649","reported_seq":249663,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:30.526082+0000","last_change":"2023-10-03T18:12:29.437740+0000","last_active":"2023-10-03T18:12:30.526082+0000","last_peered":"2023-10-03T18:12:30.526082+0000","last_clean":"2023-10-03T18:12:30.526082+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:12:30.526082+0000","last_undegraded":"2023-10-03T18:12:30.526082+0000","last_fullsized":"2023-10-03T18:12:30.526082+0000","mapping_epoch":12,"log_start":"133'123648","ondisk_log_start":"133'123648","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":50,"snaptrim_duration":3.718478781,"stat_sum":{"num_bytes":2616209980,"num_objects":4454,"num_object_clones":3851,"num_object_copies":8908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":127223,"num_read_kb":473541,"num_write":119405,"num_write_kb":476992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"1"}]},{"pgid":"2.5","version":"172'161088","reported_seq":297476,"reported_epoch":172,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:30.428710+0000","last_change":"2023-10-03T18:12:27.090664+0000","last_active":"2023-10-03T18:12:30.428710+0000","last_peered":"2023-10-03T18:12:30.428710+0000","last_clean":"2023-10-03T18:12:30.428710+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:12:30.428710+0000","last_undegraded":"2023-10-03T18:12:30.428710+0000","last_fullsized":"2023-10-03T18:12:30.428710+0000","mapping_epoch":12,"log_start":"133'151032","ondisk_log_start":"133'151032","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":75,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":40,"snaptrim_duration":5.2846337139999999,"stat_sum":{"num_bytes":3067761244,"num_objects":5674,"num_object_clones":4965,"num_object_copies":11348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5674,"num_whiteouts":0,"num_read":152830,"num_read_kb":557123,"num_write":140468,"num_write_kb":560598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"5"}]},{"pgid":"2.0","version":"172'151197","reported_seq":281503,"reported_epoch":172,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:12:28.269954+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:12:28.269954+0000","last_peered":"2023-10-03T18:12:28.269954+0000","last_clean":"2023-10-03T18:12:28.269954+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:12:28.269954+0000","last_undegraded":"2023-10-03T18:12:28.269954+0000","last_fullsized":"2023-10-03T18:12:28.269954+0000","mapping_epoch":12,"log_start":"133'141115","ondisk_log_start":"133'141115","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2544,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 283s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2964022004,"num_objects":6263,"num_object_clones":5576,"num_object_copies":12526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6263,"num_whiteouts":0,"num_read":141949,"num_read_kb":533816,"num_write":135806,"num_write_kb":542487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":329,"reported_epoch":171,"state":"active+clean","last_fresh":"2023-10-03T18:12:16.859999+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:12:16.859999+0000","last_peered":"2023-10-03T18:12:16.859999+0000","last_clean":"2023-10-03T18:12:16.859999+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:12:16.859999+0000","last_undegraded":"2023-10-03T18:12:16.859999+0000","last_fullsized":"2023-10-03T18:12:16.859999+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"172'154989","reported_seq":287326,"reported_epoch":172,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:30.386243+0000","last_change":"2023-10-03T18:12:29.273788+0000","last_active":"2023-10-03T18:12:30.386243+0000","last_peered":"2023-10-03T18:12:30.386243+0000","last_clean":"2023-10-03T18:12:30.386243+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:12:30.386243+0000","last_undegraded":"2023-10-03T18:12:30.386243+0000","last_fullsized":"2023-10-03T18:12:30.386243+0000","mapping_epoch":12,"log_start":"133'144952","ondisk_log_start":"133'144952","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":70,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":7.3978759460000001,"stat_sum":{"num_bytes":3014677372,"num_objects":5459,"num_object_clones":4759,"num_object_copies":10918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":145129,"num_read_kb":545026,"num_write":137718,"num_write_kb":550065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"}]},{"pgid":"2.1","version":"172'145026","reported_seq":265768,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:28.277085+0000","last_change":"2023-10-03T18:12:26.676378+0000","last_active":"2023-10-03T18:12:28.277085+0000","last_peered":"2023-10-03T18:12:28.277085+0000","last_clean":"2023-10-03T18:12:28.277085+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:12:28.277085+0000","last_undegraded":"2023-10-03T18:12:28.277085+0000","last_fullsized":"2023-10-03T18:12:28.277085+0000","mapping_epoch":12,"log_start":"140'134944","ondisk_log_start":"140'134944","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:10:15.175002+0000","scrub_duration":424.08306342600002,"objects_trimmed":98,"snaptrim_duration":4.8824668310000003,"stat_sum":{"num_bytes":2746651146,"num_objects":3950,"num_object_clones":3317,"num_object_copies":7900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3950,"num_whiteouts":0,"num_read":134386,"num_read_kb":494991,"num_write":125511,"num_write_kb":501031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"3"}]},{"pgid":"2.2","version":"172'146878","reported_seq":270886,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:29.700203+0000","last_change":"2023-10-03T18:12:28.838370+0000","last_active":"2023-10-03T18:12:29.700203+0000","last_peered":"2023-10-03T18:12:29.700203+0000","last_clean":"2023-10-03T18:12:29.700203+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:12:29.700203+0000","last_undegraded":"2023-10-03T18:12:29.700203+0000","last_fullsized":"2023-10-03T18:12:29.700203+0000","mapping_epoch":12,"log_start":"140'136792","ondisk_log_start":"140'136792","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:29.743761+0000","scrub_duration":370.574046178,"objects_trimmed":138,"snaptrim_duration":7.0530758899999997,"stat_sum":{"num_bytes":2750038839,"num_objects":2976,"num_object_clones":2333,"num_object_copies":5952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2976,"num_whiteouts":0,"num_read":140260,"num_read_kb":502953,"num_write":126642,"num_write_kb":505145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"2"}]},{"pgid":"2.4","version":"172'141384","reported_seq":262578,"reported_epoch":172,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:30.382956+0000","last_change":"2023-10-03T18:12:29.432608+0000","last_active":"2023-10-03T18:12:30.382956+0000","last_peered":"2023-10-03T18:12:30.382956+0000","last_clean":"2023-10-03T18:12:30.382956+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:12:30.382956+0000","last_undegraded":"2023-10-03T18:12:30.382956+0000","last_fullsized":"2023-10-03T18:12:30.382956+0000","mapping_epoch":12,"log_start":"133'131347","ondisk_log_start":"133'131347","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":7.6229011079999998,"stat_sum":{"num_bytes":2737173219,"num_objects":4751,"num_object_clones":4113,"num_object_copies":9502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4751,"num_whiteouts":0,"num_read":132896,"num_read_kb":493671,"num_write":124982,"num_write_kb":499135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"b"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22729717009,"num_objects":40765,"num_object_clones":35501,"num_object_copies":81530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40765,"num_whiteouts":0,"num_read":1111069,"num_read_kb":4113585,"num_write":1040059,"num_write_kb":4152793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8502353920,"data_stored":60437690914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80428,"ondisk_log_size":80428,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738846,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2334604,"kb_used_data":1594244,"kb_used_omap":969,"kb_used_meta":739382,"kb_avail":92037236,"statfs":{"total":96636764160,"available":94246129664,"internally_reserved":0,"allocated":1632505856,"data_stored":11627702573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":757128046},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,2,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":2.694,"15min":2.5329999999999999},"min":{"1min":0.19600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":9.2159999999999993,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.20599999999999999},{"interface":"front","average":{"1min":2.081,"5min":2.0070000000000001,"15min":2.4910000000000001},"min":{"1min":0.22,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":7.556,"5min":16.285,"15min":18.041},"last":3.452}]},{"osd":1,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.71,"15min":2.5150000000000001},"min":{"1min":0.26100000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":11.539,"5min":15.08,"15min":26.074000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.6179999999999999,"15min":2.2509999999999999},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":12.853,"5min":14.917,"15min":19.097000000000001},"last":1.23}]}]},{"osd":0,"up_from":8,"seq":34359738845,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3640072,"kb_used_data":2559936,"kb_used_omap":2419,"kb_used_meta":1077708,"kb_avail":90731768,"statfs":{"total":96636764160,"available":92909330432,"internally_reserved":0,"allocated":2621374464,"data_stored":18567298922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1103573264},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":5.2770000000000001,"5min":3.5640000000000001,"15min":3.706},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.17299999999999999},"max":{"1min":29.760999999999999,"5min":29.760999999999999,"15min":30.852},"last":0.20399999999999999},{"interface":"front","average":{"1min":4.3120000000000003,"5min":2.532,"15min":2.7069999999999999},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":27.984999999999999,"5min":27.984999999999999,"15min":27.984999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.7689999999999999,"15min":2.0150000000000001},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":11.334,"5min":12.228999999999999,"15min":14.195},"last":0.33000000000000002},{"interface":"front","average":{"1min":2.056,"5min":1.8720000000000001,"15min":2.0790000000000002},"min":{"1min":0.247,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":11.837,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.25900000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5422148,"kb_used_data":4153144,"kb_used_omap":2305,"kb_used_meta":1266686,"kb_avail":88949692,"statfs":{"total":96636764160,"available":91084484608,"internally_reserved":0,"allocated":4252819456,"data_stored":30214182884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1297087311},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,4,4,7,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":2.5,"15min":2.863},"min":{"1min":0.20599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":9.6400000000000006,"5min":13.519,"15min":16.997},"last":5.2359999999999998},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.99,"15min":3.161},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":8.7919999999999998,"5min":10.5,"15min":24.481000000000002},"last":5.1929999999999996}]},{"osd":2,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.1400000000000001,"15min":2.9100000000000001},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.419,"5min":21.326000000000001,"15min":24.579000000000001},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.9550000000000001,"15min":2.2949999999999999},"min":{"1min":0.248,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":7.9020000000000001,"5min":10.890000000000001,"15min":20.108000000000001},"last":0.84199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2619187200,"data_stored":18566593687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4252082176,"data_stored":30235587857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1631084544,"data_stored":11635509370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:12:34.451 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:34.451 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:34.451 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:34.451 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:34.452 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:34.452 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:12:34.632 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:34.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:12:34.633 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:34.632+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:12:34.777 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:34.777+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:12:34.792 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:34.788+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:12:36.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.369+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:36.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.436+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:36.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.436+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:36.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.593+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:36.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.597+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:36.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.680+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:36.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.680+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:36.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.697+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:36.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.699+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:36.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.710+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:36.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:36.759+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:37.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.677+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:37.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.712+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:37.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.712+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:37.771 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.771+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:37.771 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.771+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:37.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.779+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:37.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.779+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:37.809 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.808+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:37.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.824+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:37.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.826+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:37.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:37.826+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:39.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.267+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:39.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.305+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:39.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.305+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:39.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.305+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:39.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.317+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:39.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:39.370+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:40.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.934+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:40.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:40.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:40.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:40.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:40.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:40.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:40.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:40.940+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:42.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.169+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:42.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.202+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:42.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.202+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:42.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.221+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:42.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.230+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:42.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.237+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:42.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.417+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:42.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.417+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:42.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.458+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:42.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.458+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:42.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:42.463+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:43.623 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:43.621+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:43.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:43.658+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:43.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:43.662+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:44.818 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:44.817+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:12:46.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.707+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:46.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:46.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:46.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:46.847+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:47.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.326+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:12:47.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.326+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:47.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.327+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:47.359 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.354+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:47.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.379+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:47.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.387+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:47.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.390+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:47.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.390+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:47.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.415+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:47.417 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.415+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:47.417 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:47.415+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:48.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:48.321+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:48.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:48.432+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:51.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.435+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:51.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.435+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:51.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.456+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:51.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.456+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:51.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.456+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:51.461 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.457+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:51.461 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.457+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:51.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.457+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:51.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.571+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:51.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.584+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:51.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.584+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:51.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.635+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:51.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.658+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:51.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:51.672+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:53.989 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:53.982+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:12:54.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.223+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:54.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.223+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:54.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.299+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:54.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.299+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:54.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.325+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:54.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.325+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:54.410 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.388+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:54.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:12:54.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.479+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:54.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:54.480+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:55.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:12:55.218 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:12:55.232 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1372,"stamp":"2023-10-03T18:12:54.830030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22762174257,"num_objects":40622,"num_object_clones":35356,"num_object_copies":81244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40622,"num_whiteouts":0,"num_read":1118874,"num_read_kb":4144104,"num_write":1047598,"num_write_kb":4184295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80355,"ondisk_log_size":80355,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11690716,"kb_used_data":8365696,"kb_used_omap":5694,"kb_used_meta":3319297,"kb_avail":271424804,"statfs":{"total":289910292480,"available":277938999296,"internally_reserved":0,"allocated":8566472704,"data_stored":60423449595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3398961101},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,4,11,29,13,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14942208,"num_objects":-205,"num_object_clones":-205,"num_object_copies":-410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-205,"num_whiteouts":0,"num_read":3570,"num_read_kb":14108,"num_write":3480,"num_write_kb":13920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.040951"},"pg_stats":[{"pgid":"2.7","version":"176'145936","reported_seq":272227,"reported_epoch":176,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:52.868985+0000","last_change":"2023-10-03T18:12:51.458690+0000","last_active":"2023-10-03T18:12:52.868985+0000","last_peered":"2023-10-03T18:12:52.868985+0000","last_clean":"2023-10-03T18:12:52.868985+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:12:52.868985+0000","last_undegraded":"2023-10-03T18:12:52.868985+0000","last_fullsized":"2023-10-03T18:12:52.868985+0000","mapping_epoch":12,"log_start":"133'135888","ondisk_log_start":"133'135888","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":136,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:37:09.662642+0000","scrub_duration":505.38444223300002,"objects_trimmed":20,"snaptrim_duration":1.8854915640000001,"stat_sum":{"num_bytes":2837021157,"num_objects":7229,"num_object_clones":6578,"num_object_copies":14458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7229,"num_whiteouts":0,"num_read":137326,"num_read_kb":516160,"num_write":130404,"num_write_kb":520848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"71","length":"1"}]},{"pgid":"2.6","version":"176'134872","reported_seq":251588,"reported_epoch":176,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:52.834188+0000","last_change":"2023-10-03T18:12:51.453655+0000","last_active":"2023-10-03T18:12:52.834188+0000","last_peered":"2023-10-03T18:12:52.834188+0000","last_clean":"2023-10-03T18:12:52.834188+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:12:52.834188+0000","last_undegraded":"2023-10-03T18:12:52.834188+0000","last_fullsized":"2023-10-03T18:12:52.834188+0000","mapping_epoch":12,"log_start":"133'124853","ondisk_log_start":"133'124853","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":57,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.6236748399999996,"stat_sum":{"num_bytes":2619630140,"num_objects":4456,"num_object_clones":3853,"num_object_copies":8912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4456,"num_whiteouts":0,"num_read":128085,"num_read_kb":476933,"num_write":120231,"num_write_kb":480296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"3"}]},{"pgid":"2.5","version":"176'162529","reported_seq":299815,"reported_epoch":176,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:52.830723+0000","last_change":"2023-10-03T18:12:50.329573+0000","last_active":"2023-10-03T18:12:52.830723+0000","last_peered":"2023-10-03T18:12:52.830723+0000","last_clean":"2023-10-03T18:12:52.830723+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:12:52.830723+0000","last_undegraded":"2023-10-03T18:12:52.830723+0000","last_fullsized":"2023-10-03T18:12:52.830723+0000","mapping_epoch":12,"log_start":"140'152438","ondisk_log_start":"140'152438","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":71,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.5204427850000002,"stat_sum":{"num_bytes":3072033372,"num_objects":5684,"num_object_clones":4975,"num_object_copies":11368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":153892,"num_read_kb":561339,"num_write":141446,"num_write_kb":564510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"9"}]},{"pgid":"2.0","version":"176'152652","reported_seq":283894,"reported_epoch":176,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:12:53.473464+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:12:53.473464+0000","last_peered":"2023-10-03T18:12:53.473464+0000","last_clean":"2023-10-03T18:12:53.473464+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:12:53.473464+0000","last_undegraded":"2023-10-03T18:12:53.473464+0000","last_fullsized":"2023-10-03T18:12:53.473464+0000","mapping_epoch":12,"log_start":"133'142618","ondisk_log_start":"133'142618","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2747,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 309s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2968679156,"num_objects":6448,"num_object_clones":5761,"num_object_copies":12896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6448,"num_whiteouts":0,"num_read":143042,"num_read_kb":538112,"num_write":136882,"num_write_kb":546791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":333,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-03T18:12:46.808689+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:12:46.808689+0000","last_peered":"2023-10-03T18:12:46.808689+0000","last_clean":"2023-10-03T18:12:46.808689+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:12:46.808689+0000","last_undegraded":"2023-10-03T18:12:46.808689+0000","last_fullsized":"2023-10-03T18:12:46.808689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"176'156371","reported_seq":289606,"reported_epoch":176,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:12:52.698923+0000","last_change":"2023-10-03T18:12:51.390529+0000","last_active":"2023-10-03T18:12:52.698923+0000","last_peered":"2023-10-03T18:12:52.698923+0000","last_clean":"2023-10-03T18:12:52.698923+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:12:52.698923+0000","last_undegraded":"2023-10-03T18:12:52.698923+0000","last_fullsized":"2023-10-03T18:12:52.698923+0000","mapping_epoch":12,"log_start":"139'146353","ondisk_log_start":"139'146353","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":66,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.5819698799999999,"stat_sum":{"num_bytes":3018716028,"num_objects":5459,"num_object_clones":4759,"num_object_copies":10918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5459,"num_whiteouts":0,"num_read":146199,"num_read_kb":549274,"num_write":138663,"num_write_kb":553845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"4"}]},{"pgid":"2.1","version":"176'147072","reported_seq":268562,"reported_epoch":176,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:53.474660+0000","last_change":"2023-10-03T18:12:52.149791+0000","last_active":"2023-10-03T18:12:53.474660+0000","last_peered":"2023-10-03T18:12:53.474660+0000","last_clean":"2023-10-03T18:12:53.474660+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:12:53.474660+0000","last_undegraded":"2023-10-03T18:12:53.474660+0000","last_fullsized":"2023-10-03T18:12:53.474660+0000","mapping_epoch":12,"log_start":"146'137058","ondisk_log_start":"146'137058","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":22,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:10:15.175002+0000","scrub_duration":424.08306342600002,"objects_trimmed":34,"snaptrim_duration":0.32935261300000002,"stat_sum":{"num_bytes":2750947850,"num_objects":3614,"num_object_clones":2981,"num_object_copies":7228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3614,"num_whiteouts":0,"num_read":135396,"num_read_kb":498971,"num_write":126510,"num_write_kb":505027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"9"}]},{"pgid":"2.2","version":"175'148101","reported_seq":272736,"reported_epoch":175,"state":"active+clean","last_fresh":"2023-10-03T18:12:49.856002+0000","last_change":"2023-10-03T18:12:38.830419+0000","last_active":"2023-10-03T18:12:49.856002+0000","last_peered":"2023-10-03T18:12:49.856002+0000","last_clean":"2023-10-03T18:12:49.856002+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:12:49.856002+0000","last_undegraded":"2023-10-03T18:12:49.856002+0000","last_fullsized":"2023-10-03T18:12:49.856002+0000","mapping_epoch":12,"log_start":"144'138094","ondisk_log_start":"144'138094","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:29.743761+0000","scrub_duration":370.574046178,"objects_trimmed":110,"snaptrim_duration":7.0290492909999998,"stat_sum":{"num_bytes":2753434423,"num_objects":2970,"num_object_clones":2327,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":141049,"num_read_kb":506065,"num_write":127425,"num_write_kb":508277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"70","length":"8"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"}]},{"pgid":"2.4","version":"176'142678","reported_seq":264606,"reported_epoch":176,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:12:52.827920+0000","last_change":"2023-10-03T18:12:51.396372+0000","last_active":"2023-10-03T18:12:52.827920+0000","last_peered":"2023-10-03T18:12:52.827920+0000","last_clean":"2023-10-03T18:12:52.827920+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:12:52.827920+0000","last_undegraded":"2023-10-03T18:12:52.827920+0000","last_fullsized":"2023-10-03T18:12:52.827920+0000","mapping_epoch":12,"log_start":"133'132653","ondisk_log_start":"133'132653","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":61,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":0.76388621199999995,"stat_sum":{"num_bytes":2741121763,"num_objects":4760,"num_object_clones":4122,"num_object_copies":9520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4760,"num_whiteouts":0,"num_read":133779,"num_read_kb":497159,"num_write":125900,"num_write_kb":502807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"71","length":"7"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22761583889,"num_objects":40620,"num_object_clones":35356,"num_object_copies":81240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40620,"num_whiteouts":0,"num_read":1118768,"num_read_kb":4144013,"num_write":1047461,"num_write_kb":4182401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8561008640,"data_stored":60406471202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80256,"ondisk_log_size":80256,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2415180,"kb_used_data":1606788,"kb_used_omap":969,"kb_used_meta":807414,"kb_avail":91956660,"statfs":{"total":96636764160,"available":94163619840,"internally_reserved":0,"allocated":1645350912,"data_stored":11553559341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":826792814},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,5,6,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":1.9390000000000001,"5min":2.694,"15min":2.5329999999999999},"min":{"1min":0.19600000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":9.2159999999999993,"5min":32.173000000000002,"15min":32.173000000000002},"last":2.0099999999999998},{"interface":"front","average":{"1min":2.081,"5min":2.0070000000000001,"15min":2.4910000000000001},"min":{"1min":0.22,"5min":0.20499999999999999,"15min":0.17199999999999999},"max":{"1min":7.556,"5min":16.285,"15min":18.041},"last":0.219}]},{"osd":1,"last update":"Tue Oct 3 18:12:01 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.71,"15min":2.5150000000000001},"min":{"1min":0.26100000000000001,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":11.539,"5min":15.08,"15min":26.074000000000002},"last":2.8010000000000002},{"interface":"front","average":{"1min":2.7429999999999999,"5min":2.6179999999999999,"15min":2.2509999999999999},"min":{"1min":0.23499999999999999,"5min":0.188,"15min":0.188},"max":{"1min":12.853,"5min":14.917,"15min":19.097000000000001},"last":2.9670000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738849,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3701288,"kb_used_data":2575776,"kb_used_omap":2419,"kb_used_meta":1123084,"kb_avail":90670552,"statfs":{"total":96636764160,"available":92846645248,"internally_reserved":0,"allocated":2637594624,"data_stored":18657482474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1150038288},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":5.2770000000000001,"5min":3.5640000000000001,"15min":3.706},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.17299999999999999},"max":{"1min":29.760999999999999,"5min":29.760999999999999,"15min":30.852},"last":0.46899999999999997},{"interface":"front","average":{"1min":4.3120000000000003,"5min":2.532,"15min":2.7069999999999999},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":27.984999999999999,"5min":27.984999999999999,"15min":27.984999999999999},"last":1.506}]},{"osd":2,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.7689999999999999,"15min":2.0150000000000001},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":11.334,"5min":12.228999999999999,"15min":14.195},"last":7.5549999999999997},{"interface":"front","average":{"1min":2.056,"5min":1.8720000000000001,"15min":2.0790000000000002},"min":{"1min":0.247,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":11.837,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.316}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5574248,"kb_used_data":4183132,"kb_used_omap":2305,"kb_used_meta":1388798,"kb_avail":88797592,"statfs":{"total":96636764160,"available":90928734208,"internally_reserved":0,"allocated":4283527168,"data_stored":30212407780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1422129999},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,7,23,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":2.5,"15min":2.863},"min":{"1min":0.20599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":9.6400000000000006,"5min":13.519,"15min":16.997},"last":0.33700000000000002},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.99,"15min":3.161},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":8.7919999999999998,"5min":10.5,"15min":24.481000000000002},"last":1.2969999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.1400000000000001,"15min":2.9100000000000001},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.419,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.9550000000000001,"15min":2.2949999999999999},"min":{"1min":0.248,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":7.9020000000000001,"5min":10.890000000000001,"15min":20.108000000000001},"last":1.343}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2635411456,"data_stored":18657991831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4281688064,"data_stored":30198846737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1643909120,"data_stored":11549632634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:12:55.233 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:55.234 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:55.234 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:55.234 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:55.234 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:12:55.235 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:12:56.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.170+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:56.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.170+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:56.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.458+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:56.500 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.494+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:56.500 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.494+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:56.526 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.524+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:56.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.531+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:56.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:56.531+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:57.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.336+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:12:57.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.345+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:57.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.345+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:57.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.345+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:57.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.354+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:57.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.382+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:57.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.382+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:12:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.398+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.398+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:57.423 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:57.418+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:12:57.842 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:57.840+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:12:57.842 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:57.840+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:12:57.842 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:57.840+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:12:57.843 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:57.840+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:12:57.843 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:12:57.840+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:12:59.441 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.440+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:12:59.442 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.440+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:12:59.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.479+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:12:59.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.479+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:12:59.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.480+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:12:59.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.480+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:12:59.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.525+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:12:59.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:12:59.529+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:00.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:00.693+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:00.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:00.728+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:00.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:00.751+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:03.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:03.146+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:03.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:03.166+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:04.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.551+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:04.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.551+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:04.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.607+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:04.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.748+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:04.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.757+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:04.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:04.992+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:05.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.008+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:05.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.100+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:05.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.100+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:05.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.130+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:05.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.161+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:05.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:05.180+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:06.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.186+0000 7fc515f98700 -1 reset not still connected to 0x56222214f040 2023-10-03T18:13:06.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.257+0000 7fc515f98700 -1 reset not still connected to 0x562236499860 2023-10-03T18:13:06.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.581+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:06.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.641+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:06.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.641+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:06.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.685+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:06.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.685+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:06.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.727+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:06.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.753+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:06.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:06.850+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:07.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.734+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:07.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.753+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:07.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:07.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:07.820 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.818+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:07.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.863+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:07.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.863+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:07.921 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.918+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:07.921 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:07.918+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:10.140 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.140+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:10.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.219+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:10.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.219+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:10.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.219+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:10.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.250+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:10.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.251+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:10.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.251+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:10.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.299+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:10.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.300+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:10.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:10.351+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:13.279 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:13.279+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:13.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:13.418+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:13.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:13.418+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:13.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:13.418+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:13.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:13.492+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:14.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.433+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:14.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.520+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:14.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.520+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:14.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.520+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:14.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.524+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:14.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:14.700+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:15.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:13:15.900 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:13:15.916 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:13:15.944 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1393,"stamp":"2023-10-03T18:13:14.839634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22784567089,"num_objects":41186,"num_object_clones":35920,"num_object_copies":82372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41186,"num_whiteouts":0,"num_read":1127094,"num_read_kb":4167061,"num_write":1053101,"num_write_kb":4205587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80449,"ondisk_log_size":80449,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11377700,"kb_used_data":8409784,"kb_used_omap":5694,"kb_used_meta":2962177,"kb_avail":271737820,"statfs":{"total":289910292480,"available":278259527680,"internally_reserved":0,"allocated":8611618816,"data_stored":61326150953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3033270221},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,9,6,6,11,10,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6741863,"num_objects":726,"num_object_clones":729,"num_object_copies":1452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":2553,"num_read_kb":7033,"num_write":1736,"num_write_kb":6612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007007"},"pg_stats":[{"pgid":"2.7","version":"187'146906","reported_seq":273728,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:09.709214+0000","last_change":"2023-10-03T18:13:07.334849+0000","last_active":"2023-10-03T18:13:09.709214+0000","last_peered":"2023-10-03T18:13:09.709214+0000","last_clean":"2023-10-03T18:13:09.709214+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:13:09.709214+0000","last_undegraded":"2023-10-03T18:13:09.709214+0000","last_fullsized":"2023-10-03T18:13:09.709214+0000","mapping_epoch":12,"log_start":"140'136892","ondisk_log_start":"140'136892","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":153,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:21:00.491363+0000","scrub_duration":505.38444223300002,"objects_trimmed":8,"snaptrim_duration":0.151077829,"stat_sum":{"num_bytes":2839634558,"num_objects":7324,"num_object_clones":6670,"num_object_copies":14648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7324,"num_whiteouts":0,"num_read":138244,"num_read_kb":518876,"num_write":131035,"num_write_kb":523291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"187'135783","reported_seq":253025,"reported_epoch":187,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:09.572208+0000","last_change":"2023-10-03T18:13:07.439348+0000","last_active":"2023-10-03T18:13:09.572208+0000","last_peered":"2023-10-03T18:13:09.572208+0000","last_clean":"2023-10-03T18:13:09.572208+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:13:09.572208+0000","last_undegraded":"2023-10-03T18:13:09.572208+0000","last_fullsized":"2023-10-03T18:13:09.572208+0000","mapping_epoch":12,"log_start":"133'125760","ondisk_log_start":"133'125760","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.52238604300000002,"stat_sum":{"num_bytes":2622075401,"num_objects":4539,"num_object_clones":3937,"num_object_copies":9078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4539,"num_whiteouts":0,"num_read":128884,"num_read_kb":479397,"num_write":120819,"num_write_kb":482615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"6"}]},{"pgid":"2.5","version":"187'163647","reported_seq":301566,"reported_epoch":187,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:09.476651+0000","last_change":"2023-10-03T18:13:06.295093+0000","last_active":"2023-10-03T18:13:09.476651+0000","last_peered":"2023-10-03T18:13:09.476651+0000","last_clean":"2023-10-03T18:13:09.476651+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:13:09.476651+0000","last_undegraded":"2023-10-03T18:13:09.476651+0000","last_fullsized":"2023-10-03T18:13:09.476651+0000","mapping_epoch":12,"log_start":"142'153641","ondisk_log_start":"142'153641","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":88,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34041514,"stat_sum":{"num_bytes":3074695874,"num_objects":5766,"num_object_clones":5055,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5766,"num_whiteouts":0,"num_read":155151,"num_read_kb":564483,"num_write":142138,"num_write_kb":567135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"2"}]},{"pgid":"2.0","version":"187'153876","reported_seq":285650,"reported_epoch":187,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:13:13.771971+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:13:13.771971+0000","last_peered":"2023-10-03T18:13:13.771971+0000","last_clean":"2023-10-03T18:13:13.771971+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:13:13.771971+0000","last_undegraded":"2023-10-03T18:13:13.771971+0000","last_fullsized":"2023-10-03T18:13:13.771971+0000","mapping_epoch":12,"log_start":"140'143820","ondisk_log_start":"140'143820","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":2944,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 329s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2971968195,"num_objects":6701,"num_object_clones":6015,"num_object_copies":13402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6701,"num_whiteouts":0,"num_read":144019,"num_read_kb":541085,"num_write":137669,"num_write_kb":549863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":344,"reported_epoch":186,"state":"active+clean","last_fresh":"2023-10-03T18:13:07.898991+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:13:07.898991+0000","last_peered":"2023-10-03T18:13:07.898991+0000","last_clean":"2023-10-03T18:13:07.898991+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:13:07.898991+0000","last_undegraded":"2023-10-03T18:13:07.898991+0000","last_fullsized":"2023-10-03T18:13:07.898991+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"187'157454","reported_seq":291191,"reported_epoch":187,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:09.404360+0000","last_change":"2023-10-03T18:13:07.211114+0000","last_active":"2023-10-03T18:13:09.404360+0000","last_peered":"2023-10-03T18:13:09.404360+0000","last_clean":"2023-10-03T18:13:09.404360+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:13:09.404360+0000","last_undegraded":"2023-10-03T18:13:09.404360+0000","last_fullsized":"2023-10-03T18:13:09.404360+0000","mapping_epoch":12,"log_start":"141'147355","ondisk_log_start":"141'147355","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":83,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.35398519499999997,"stat_sum":{"num_bytes":3021615894,"num_objects":5531,"num_object_clones":4833,"num_object_copies":11062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5531,"num_whiteouts":0,"num_read":147101,"num_read_kb":551928,"num_write":139353,"num_write_kb":556525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"}]},{"pgid":"2.1","version":"187'148738","reported_seq":270666,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:13.756796+0000","last_change":"2023-10-03T18:13:13.467197+0000","last_active":"2023-10-03T18:13:13.756796+0000","last_peered":"2023-10-03T18:13:13.756796+0000","last_clean":"2023-10-03T18:13:13.756796+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:13:13.756796+0000","last_undegraded":"2023-10-03T18:13:13.756796+0000","last_fullsized":"2023-10-03T18:13:13.756796+0000","mapping_epoch":12,"log_start":"156'138672","ondisk_log_start":"156'138672","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":30,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:08:29.548871+0000","scrub_duration":424.08306342600002,"objects_trimmed":71,"snaptrim_duration":4.5516612209999998,"stat_sum":{"num_bytes":2753974793,"num_objects":3500,"num_object_clones":2867,"num_object_copies":7000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3500,"num_whiteouts":0,"num_read":136481,"num_read_kb":501925,"num_write":127247,"num_write_kb":507881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"}]},{"pgid":"2.2","version":"187'149280","reported_seq":274656,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:09.963061+0000","last_change":"2023-10-03T18:13:09.549160+0000","last_active":"2023-10-03T18:13:09.963061+0000","last_peered":"2023-10-03T18:13:09.963061+0000","last_clean":"2023-10-03T18:13:09.963061+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:13:09.963061+0000","last_undegraded":"2023-10-03T18:13:09.963061+0000","last_fullsized":"2023-10-03T18:13:09.963061+0000","mapping_epoch":12,"log_start":"147'139201","ondisk_log_start":"147'139201","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:34:17.311310+0000","scrub_duration":370.574046178,"objects_trimmed":16,"snaptrim_duration":0.64159171400000004,"stat_sum":{"num_bytes":2756391735,"num_objects":3006,"num_object_clones":2363,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":142400,"num_read_kb":509508,"num_write":128177,"num_write_kb":511148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"4"}]},{"pgid":"2.4","version":"187'143664","reported_seq":266138,"reported_epoch":187,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:09.516195+0000","last_change":"2023-10-03T18:13:07.443678+0000","last_active":"2023-10-03T18:13:09.516195+0000","last_peered":"2023-10-03T18:13:09.516195+0000","last_clean":"2023-10-03T18:13:09.516195+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:13:09.516195+0000","last_undegraded":"2023-10-03T18:13:09.516195+0000","last_fullsized":"2023-10-03T18:13:09.516195+0000","mapping_epoch":12,"log_start":"140'133657","ondisk_log_start":"140'133657","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":78,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.66134271200000005,"stat_sum":{"num_bytes":2743620271,"num_objects":4817,"num_object_clones":4180,"num_object_copies":9634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4817,"num_whiteouts":0,"num_read":134708,"num_read_kb":499768,"num_write":126526,"num_write_kb":505235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"75","length":"3"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22783976721,"num_objects":41184,"num_object_clones":35920,"num_object_copies":82368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41184,"num_whiteouts":0,"num_read":1126988,"num_read_kb":4166970,"num_write":1052964,"num_write_kb":4203693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8605769728,"data_stored":61430364706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80350,"ondisk_log_size":80350,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2452080,"kb_used_data":1615912,"kb_used_omap":969,"kb_used_meta":835190,"kb_avail":91919760,"statfs":{"total":96636764160,"available":94125834240,"internally_reserved":0,"allocated":1654693888,"data_stored":11793800399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":855235438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":2.669,"15min":2.488},"min":{"1min":0.17899999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":6.984,"5min":32.173000000000002,"15min":32.173000000000002},"last":3.9129999999999998},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.8959999999999999,"15min":2.4390000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":10.994999999999999,"5min":12.731999999999999,"15min":18.041},"last":0.253}]},{"osd":1,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.823,"5min":2.7509999999999999,"15min":2.5579999999999998},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":18.768999999999998,"5min":18.768999999999998,"15min":26.074000000000002},"last":4.1890000000000001},{"interface":"front","average":{"1min":2.9590000000000001,"5min":2.7909999999999999,"15min":2.3300000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":18.751999999999999,"5min":18.751999999999999,"15min":19.097000000000001},"last":4.1710000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3267576,"kb_used_data":2590824,"kb_used_omap":2419,"kb_used_meta":674316,"kb_avail":91104264,"statfs":{"total":96636764160,"available":93290766336,"internally_reserved":0,"allocated":2653003776,"data_stored":18972496956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":690499856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":5.2770000000000001,"5min":3.5640000000000001,"15min":3.706},"min":{"1min":0.20300000000000001,"5min":0.191,"15min":0.17299999999999999},"max":{"1min":29.760999999999999,"5min":29.760999999999999,"15min":30.852},"last":0.19},{"interface":"front","average":{"1min":4.3120000000000003,"5min":2.532,"15min":2.7069999999999999},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":27.984999999999999,"5min":27.984999999999999,"15min":27.984999999999999},"last":11.291}]},{"osd":2,"last update":"Tue Oct 3 18:12:14 2023","interfaces":[{"interface":"back","average":{"1min":2.0659999999999998,"5min":1.7689999999999999,"15min":2.0150000000000001},"min":{"1min":0.214,"5min":0.182,"15min":0.182},"max":{"1min":11.334,"5min":12.228999999999999,"15min":14.195},"last":3.7280000000000002},{"interface":"front","average":{"1min":2.056,"5min":1.8720000000000001,"15min":2.0790000000000002},"min":{"1min":0.247,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":11.837,"5min":12.880000000000001,"15min":12.880000000000001},"last":6.2220000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5658044,"kb_used_data":4203048,"kb_used_omap":2305,"kb_used_meta":1452670,"kb_avail":88713796,"statfs":{"total":96636764160,"available":90842927104,"internally_reserved":0,"allocated":4303921152,"data_stored":30559853598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1487534927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,7,5,5,10,9,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":3.125,"5min":2.5,"15min":2.863},"min":{"1min":0.20599999999999999,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":9.6400000000000006,"5min":13.519,"15min":16.997},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.99,"15min":3.161},"min":{"1min":0.20899999999999999,"5min":0.19,"15min":0.17899999999999999},"max":{"1min":8.7919999999999998,"5min":10.5,"15min":24.481000000000002},"last":1.4219999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:12:04 2023","interfaces":[{"interface":"back","average":{"1min":2.1659999999999999,"5min":2.1400000000000001,"15min":2.9100000000000001},"min":{"1min":0.20799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":12.419,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.9990000000000001,"5min":1.9550000000000001,"15min":2.2949999999999999},"min":{"1min":0.248,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":7.9020000000000001,"5min":10.890000000000001,"15min":20.108000000000001},"last":1.377}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2650710016,"data_stored":18986102063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4302049280,"data_stored":30630339857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1653010432,"data_stored":11813922786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:13:15.945 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:15.945 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:15.946 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:15.946 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:15.946 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:15.946 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:13:23.035 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.035+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:23.035 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.035+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:23.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.063+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:23.123 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.122+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:23.123 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.122+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:23.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.351+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:23.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.379+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:23.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:23.379+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:23.721 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:23.720+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:13:23.721 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:23.721+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:13:23.722 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:23.721+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:13:25.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.605+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:25.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.614+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:25.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.614+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:25.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.627+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:25.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.650+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:25.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:25.650+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:26.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.104+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:26.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.123+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:26.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.123+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:26.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:26.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.161+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:26.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.265+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:26.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.265+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:26.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.265+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:26.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.268+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:26.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.307+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:26.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:26.320+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:30.978 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:30.971+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:30.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:30.989+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:30.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:30.989+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:30.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:30.989+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:31.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.025+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:31.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.221+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:31.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.235+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:31.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.556+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:31.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.556+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:31.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.556+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:31.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.652+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:31.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.718+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:31.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.718+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:31.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.800+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:31.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.801+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:31.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.801+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:31.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.809+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:31.825 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:31.824+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:32.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.150+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:32.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.219+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:32.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.219+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:32.248 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.242+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:32.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.314+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:32.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.315+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:32.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.315+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:32.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.315+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:32.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.352+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:32.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:32.352+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:35.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:13:36.711 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:13:36.716 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:13:36.737 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1404,"stamp":"2023-10-03T18:13:34.847160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22807521073,"num_objects":42607,"num_object_clones":37341,"num_object_copies":85214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42607,"num_whiteouts":0,"num_read":1133772,"num_read_kb":4188988,"num_write":1058629,"num_write_kb":4227699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80427,"ondisk_log_size":80427,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":11691872,"kb_used_data":8453256,"kb_used_omap":5694,"kb_used_meta":3232897,"kb_avail":271423648,"statfs":{"total":289910292480,"available":277937815552,"internally_reserved":0,"allocated":8656134144,"data_stored":63929931257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3310487501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,2,3,13,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13385728,"num_objects":478,"num_object_clones":478,"num_object_copies":956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":478,"num_whiteouts":0,"num_read":4513,"num_read_kb":13347,"num_write":3254,"num_write_kb":13016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"2.7","version":"188'148155","reported_seq":275285,"reported_epoch":188,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:31.543639+0000","last_change":"2023-10-03T18:13:27.616985+0000","last_active":"2023-10-03T18:13:31.543639+0000","last_peered":"2023-10-03T18:13:31.543639+0000","last_clean":"2023-10-03T18:13:31.543639+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:13:31.543639+0000","last_undegraded":"2023-10-03T18:13:31.543639+0000","last_fullsized":"2023-10-03T18:13:31.543639+0000","mapping_epoch":12,"log_start":"143'138097","ondisk_log_start":"143'138097","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":150,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:21:00.491363+0000","scrub_duration":505.38444223300002,"objects_trimmed":36,"snaptrim_duration":5.7542547749999997,"stat_sum":{"num_bytes":2842641022,"num_objects":7562,"num_object_clones":6908,"num_object_copies":15124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7562,"num_whiteouts":0,"num_read":139014,"num_read_kb":521512,"num_write":131744,"num_write_kb":526127,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"77","length":"1"}]},{"pgid":"2.6","version":"188'137002","reported_seq":254614,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:31.549777+0000","last_change":"2023-10-03T18:13:30.300174+0000","last_active":"2023-10-03T18:13:31.549777+0000","last_peered":"2023-10-03T18:13:31.549777+0000","last_clean":"2023-10-03T18:13:31.549777+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:13:31.549777+0000","last_undegraded":"2023-10-03T18:13:31.549777+0000","last_fullsized":"2023-10-03T18:13:31.549777+0000","mapping_epoch":12,"log_start":"140'126962","ondisk_log_start":"140'126962","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":71,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":48,"snaptrim_duration":4.8736215180000002,"stat_sum":{"num_bytes":2624930313,"num_objects":4743,"num_object_clones":4141,"num_object_copies":9486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4743,"num_whiteouts":0,"num_read":129691,"num_read_kb":482150,"num_write":121497,"num_write_kb":485327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"9"}]},{"pgid":"2.5","version":"188'165038","reported_seq":303404,"reported_epoch":188,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:31.520877+0000","last_change":"2023-10-03T18:13:30.297378+0000","last_active":"2023-10-03T18:13:31.520877+0000","last_peered":"2023-10-03T18:13:31.520877+0000","last_clean":"2023-10-03T18:13:31.520877+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:13:31.520877+0000","last_undegraded":"2023-10-03T18:13:31.520877+0000","last_fullsized":"2023-10-03T18:13:31.520877+0000","mapping_epoch":12,"log_start":"147'154947","ondisk_log_start":"147'154947","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":85,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":43,"snaptrim_duration":8.4184017220000005,"stat_sum":{"num_bytes":3077878466,"num_objects":6021,"num_object_clones":5310,"num_object_copies":12042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6021,"num_whiteouts":0,"num_read":156201,"num_read_kb":567811,"num_write":142913,"num_write_kb":570235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"5"}]},{"pgid":"2.0","version":"188'155003","reported_seq":287188,"reported_epoch":188,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:13:33.889228+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:13:33.889228+0000","last_peered":"2023-10-03T18:13:33.889228+0000","last_clean":"2023-10-03T18:13:33.889228+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:13:33.889228+0000","last_undegraded":"2023-10-03T18:13:33.889228+0000","last_fullsized":"2023-10-03T18:13:33.889228+0000","mapping_epoch":12,"log_start":"143'144922","ondisk_log_start":"143'144922","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3105,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 349s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2974839491,"num_objects":6987,"num_object_clones":6301,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6987,"num_whiteouts":0,"num_read":144806,"num_read_kb":543785,"num_write":138365,"num_write_kb":552647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":345,"reported_epoch":187,"state":"active+clean","last_fresh":"2023-10-03T18:13:08.948469+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:13:08.948469+0000","last_peered":"2023-10-03T18:13:08.948469+0000","last_clean":"2023-10-03T18:13:08.948469+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:13:08.948469+0000","last_undegraded":"2023-10-03T18:13:08.948469+0000","last_fullsized":"2023-10-03T18:13:08.948469+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"188'158770","reported_seq":292793,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:31.375480+0000","last_change":"2023-10-03T18:13:29.445932+0000","last_active":"2023-10-03T18:13:31.375480+0000","last_peered":"2023-10-03T18:13:31.375480+0000","last_clean":"2023-10-03T18:13:31.375480+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:13:31.375480+0000","last_undegraded":"2023-10-03T18:13:31.375480+0000","last_fullsized":"2023-10-03T18:13:31.375480+0000","mapping_epoch":12,"log_start":"146'148763","ondisk_log_start":"146'148763","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":81,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":61,"snaptrim_duration":2.917112769,"stat_sum":{"num_bytes":3024700182,"num_objects":5786,"num_object_clones":5088,"num_object_copies":11572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5786,"num_whiteouts":0,"num_read":147883,"num_read_kb":554608,"num_write":140096,"num_write_kb":559497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"}]},{"pgid":"2.1","version":"188'150312","reported_seq":272513,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:33.888020+0000","last_change":"2023-10-03T18:13:33.100708+0000","last_active":"2023-10-03T18:13:33.888020+0000","last_peered":"2023-10-03T18:13:33.888020+0000","last_clean":"2023-10-03T18:13:33.888020+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:13:33.888020+0000","last_undegraded":"2023-10-03T18:13:33.888020+0000","last_fullsized":"2023-10-03T18:13:33.888020+0000","mapping_epoch":12,"log_start":"157'140285","ondisk_log_start":"157'140285","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:08:29.548871+0000","scrub_duration":424.08306342600002,"objects_trimmed":215,"snaptrim_duration":11.253670471,"stat_sum":{"num_bytes":2756600329,"num_objects":3425,"num_object_clones":2792,"num_object_copies":6850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3425,"num_whiteouts":0,"num_read":137287,"num_read_kb":504487,"num_write":127888,"num_write_kb":510445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"1"}]},{"pgid":"2.2","version":"188'150827","reported_seq":276439,"reported_epoch":188,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:30.054442+0000","last_change":"2023-10-03T18:13:29.634590+0000","last_active":"2023-10-03T18:13:30.054442+0000","last_peered":"2023-10-03T18:13:30.054442+0000","last_clean":"2023-10-03T18:13:30.054442+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:13:30.054442+0000","last_undegraded":"2023-10-03T18:13:30.054442+0000","last_fullsized":"2023-10-03T18:13:30.054442+0000","mapping_epoch":12,"log_start":"156'140811","ondisk_log_start":"156'140811","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:34:17.311310+0000","scrub_duration":370.574046178,"objects_trimmed":127,"snaptrim_duration":7.7691412580000003,"stat_sum":{"num_bytes":2758972215,"num_objects":3032,"num_object_clones":2389,"num_object_copies":6064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3032,"num_whiteouts":0,"num_read":143277,"num_read_kb":512079,"num_write":128802,"num_write_kb":513648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"4"}]},{"pgid":"2.4","version":"188'144871","reported_seq":267688,"reported_epoch":188,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:31.542746+0000","last_change":"2023-10-03T18:13:29.445875+0000","last_active":"2023-10-03T18:13:31.542746+0000","last_peered":"2023-10-03T18:13:31.542746+0000","last_clean":"2023-10-03T18:13:31.542746+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:13:31.542746+0000","last_undegraded":"2023-10-03T18:13:31.542746+0000","last_fullsized":"2023-10-03T18:13:31.542746+0000","mapping_epoch":12,"log_start":"143'134863","ondisk_log_start":"143'134863","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":62,"snaptrim_duration":7.5824350689999997,"stat_sum":{"num_bytes":2746368687,"num_objects":5049,"num_object_clones":4412,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5049,"num_whiteouts":0,"num_read":135507,"num_read_kb":502465,"num_write":127187,"num_write_kb":507879,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"77","length":"1"},{"start":"82","length":"9"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22806930705,"num_objects":42605,"num_object_clones":37341,"num_object_copies":85210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42605,"num_whiteouts":0,"num_read":1133666,"num_read_kb":4188897,"num_write":1058492,"num_write_kb":4225805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8650436608,"data_stored":63957801506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80328,"ondisk_log_size":80328,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738859,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2504768,"kb_used_data":1624184,"kb_used_omap":969,"kb_used_meta":879606,"kb_avail":91867072,"statfs":{"total":96636764160,"available":94071881728,"internally_reserved":0,"allocated":1663164416,"data_stored":12184620991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":900717422},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":2.669,"15min":2.488},"min":{"1min":0.17899999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":6.984,"5min":32.173000000000002,"15min":32.173000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.8959999999999999,"15min":2.4390000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":10.994999999999999,"5min":12.731999999999999,"15min":18.041},"last":0.30299999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.823,"5min":2.7509999999999999,"15min":2.5579999999999998},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":18.768999999999998,"5min":18.768999999999998,"15min":26.074000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":2.9590000000000001,"5min":2.7909999999999999,"15min":2.3300000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":18.751999999999999,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.443}]}]},{"osd":0,"up_from":8,"seq":34359738857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3348392,"kb_used_data":2603616,"kb_used_omap":2419,"kb_used_meta":742348,"kb_avail":91023448,"statfs":{"total":96636764160,"available":93208010752,"internally_reserved":0,"allocated":2666102784,"data_stored":19778938668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":760164624},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":3.6920000000000002,"15min":3.649},"min":{"1min":0.19,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":17.138999999999999,"5min":29.760999999999999,"15min":30.852},"last":9.4149999999999991},{"interface":"front","average":{"1min":3.2799999999999998,"5min":2.7559999999999998,"15min":2.6749999999999998},"min":{"1min":0.22700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":11.291,"5min":27.984999999999999,"15min":27.984999999999999},"last":9.4760000000000009}]},{"osd":2,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.9359999999999999,"15min":1.907},"min":{"1min":0.249,"5min":0.182,"15min":0.182},"max":{"1min":8.0830000000000002,"5min":12.228999999999999,"15min":14.195},"last":1.8640000000000001},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.873,"15min":2.0289999999999999},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":7.9050000000000002,"5min":12.880000000000001,"15min":12.880000000000001},"last":0.39600000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738843,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5838712,"kb_used_data":4225456,"kb_used_omap":2305,"kb_used_meta":1610942,"kb_avail":88533128,"statfs":{"total":96636764160,"available":90657923072,"internally_reserved":0,"allocated":4326866944,"data_stored":31966371598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1649605455},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,3,12,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:13:09 2023","interfaces":[{"interface":"back","average":{"1min":2.1499999999999999,"5min":2.5219999999999998,"15min":2.8410000000000002},"min":{"1min":0.193,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":7.1639999999999997,"5min":13.519,"15min":16.997},"last":2.1160000000000001},{"interface":"front","average":{"1min":3.5750000000000002,"5min":2.1560000000000001,"15min":3.194},"min":{"1min":0.26900000000000002,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":13.516999999999999,"5min":13.516999999999999,"15min":24.481000000000002},"last":0.97199999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:13:09 2023","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":2.367,"15min":2.8740000000000001},"min":{"1min":0.29799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":16.135999999999999,"5min":21.326000000000001,"15min":24.579000000000001},"last":1.526},{"interface":"front","average":{"1min":3.9169999999999998,"5min":2.3109999999999999,"15min":2.3180000000000001},"min":{"1min":0.307,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":13.581,"5min":13.581,"15min":20.108000000000001},"last":1.028}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2663788544,"data_stored":19783417135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4325203968,"data_stored":31985157393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1661444096,"data_stored":12189226978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:13:36.739 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:36.739 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:36.739 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:36.739 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:36.740 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:36.740 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:13:36.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:36.946+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:36.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:36.971+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:36.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:36.971+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:36.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:36.971+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:36.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:36.971+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:37.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.004+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:37.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.020+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:37.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.021+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:37.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.088+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:37.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.103+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:37.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.115+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:37.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.690+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:37.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.690+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:37.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.690+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:37.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.690+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:37.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.698+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:37.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:37.703+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:38.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.398+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:38.402 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.402+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:38.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.492+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:38.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.492+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:38.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.493+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:38.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.538+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:38.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:38.538+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:40.011 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:40.011+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:13:40.029 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:40.029+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:13:40.050 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:40.050+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:13:42.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.437+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:42.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.557+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:42.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.557+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:42.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.557+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:42.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.558+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:42.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.647+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:42.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:42.781+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:45.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.583+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:45.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.594+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:45.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.613+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:45.691 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.690+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:45.691 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.690+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:45.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.699+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:45.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.699+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:45.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.700+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:45.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.723+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:45.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:45.781+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:49.764 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:49.764+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:13:49.798 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:49.795+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:13:49.798 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:13:49.795+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:13:51.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.004+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:51.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.010+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:51.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.010+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:51.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.021+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:51.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.435+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:51.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.479+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:51.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.479+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:51.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.514+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:51.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.540+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:51.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.634+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:51.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.634+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:51.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.653+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:51.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.653+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:51.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:51.703+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:52.889 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:52.882+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:52.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:52.882+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:52.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:52.901+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:52.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:52.903+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:53.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.022+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:53.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.022+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:53.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.034+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:53.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.034+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:53.078 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.072+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:53.079 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.072+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:53.079 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.072+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:53.098 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.093+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:13:53.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.098+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:53.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.126+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:53.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.175+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:53.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:53.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:53.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:53.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:53.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:53.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:53.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:53.180+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:55.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.042+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:55.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.042+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:55.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.069+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:55.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.131+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:55.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.172+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:55.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.172+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:55.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.203+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:55.205 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.203+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:55.263 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:55.262+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:56.741 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:13:57.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.046+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:13:57.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.073+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:13:57.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.073+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:13:57.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.073+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:13:57.114 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.113+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:13:57.114 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:57.113+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:13:57.500 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:13:57.516 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:13:57.529 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1420,"stamp":"2023-10-03T18:13:56.865207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22836524849,"num_objects":42397,"num_object_clones":37131,"num_object_copies":84794,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42397,"num_whiteouts":0,"num_read":1140662,"num_read_kb":4216272,"num_write":1065673,"num_write_kb":4255875,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80557,"ondisk_log_size":80557,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12017988,"kb_used_data":8512168,"kb_used_omap":5694,"kb_used_meta":3500097,"kb_avail":271097532,"statfs":{"total":289910292480,"available":277603872768,"internally_reserved":0,"allocated":8716460032,"data_stored":63786932345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3584100301},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,8,16,13,17,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10219520,"num_objects":-157,"num_object_clones":-157,"num_object_copies":-314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-157,"num_whiteouts":0,"num_read":2462,"num_read_kb":9752,"num_write":2480,"num_write_kb":9920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.010528"},"pg_stats":[{"pgid":"2.7","version":"192'149447","reported_seq":277257,"reported_epoch":192,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:52.768124+0000","last_change":"2023-10-03T18:13:51.777376+0000","last_active":"2023-10-03T18:13:52.768124+0000","last_peered":"2023-10-03T18:13:52.768124+0000","last_clean":"2023-10-03T18:13:52.768124+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:13:52.768124+0000","last_undegraded":"2023-10-03T18:13:52.768124+0000","last_fullsized":"2023-10-03T18:13:52.768124+0000","mapping_epoch":12,"log_start":"148'139403","ondisk_log_start":"148'139403","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":145,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:21:00.491363+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":4.8893589049999999,"stat_sum":{"num_bytes":2846151294,"num_objects":7532,"num_object_clones":6878,"num_object_copies":15064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7532,"num_whiteouts":0,"num_read":139854,"num_read_kb":524832,"num_write":132593,"num_write_kb":529523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"85","length":"2"}]},{"pgid":"2.6","version":"192'138153","reported_seq":256408,"reported_epoch":192,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:52.761157+0000","last_change":"2023-10-03T18:13:50.039464+0000","last_active":"2023-10-03T18:13:52.761157+0000","last_peered":"2023-10-03T18:13:52.761157+0000","last_clean":"2023-10-03T18:13:52.761157+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:13:52.761157+0000","last_undegraded":"2023-10-03T18:13:52.761157+0000","last_fullsized":"2023-10-03T18:13:52.761157+0000","mapping_epoch":12,"log_start":"144'128064","ondisk_log_start":"144'128064","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":66,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.1462594340000001,"stat_sum":{"num_bytes":2628166153,"num_objects":4715,"num_object_clones":4113,"num_object_copies":9430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4715,"num_whiteouts":0,"num_read":130467,"num_read_kb":485218,"num_write":122279,"num_write_kb":488455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"2"}]},{"pgid":"2.5","version":"192'166409","reported_seq":305575,"reported_epoch":192,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:13:52.787171+0000","last_change":"2023-10-03T18:13:52.628850+0000","last_active":"2023-10-03T18:13:52.787171+0000","last_peered":"2023-10-03T18:13:52.787171+0000","last_clean":"2023-10-03T18:13:52.787171+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:13:52.787171+0000","last_undegraded":"2023-10-03T18:13:52.787171+0000","last_fullsized":"2023-10-03T18:13:52.787171+0000","mapping_epoch":12,"log_start":"156'156352","ondisk_log_start":"156'156352","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":81,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.74878589200000001,"stat_sum":{"num_bytes":3081847490,"num_objects":6032,"num_object_clones":5321,"num_object_copies":12064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":157149,"num_read_kb":571575,"num_write":143882,"num_write_kb":574111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"}]},{"pgid":"2.0","version":"192'156085","reported_seq":289053,"reported_epoch":192,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:13:54.004798+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:13:54.004798+0000","last_peered":"2023-10-03T18:13:54.004798+0000","last_clean":"2023-10-03T18:13:54.004798+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:13:54.004798+0000","last_undegraded":"2023-10-03T18:13:54.004798+0000","last_fullsized":"2023-10-03T18:13:54.004798+0000","mapping_epoch":12,"log_start":"147'146024","ondisk_log_start":"147'146024","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3243,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":120,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 369s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2978316995,"num_objects":7084,"num_object_clones":6398,"num_object_copies":14168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7084,"num_whiteouts":0,"num_read":145647,"num_read_kb":547121,"num_write":139214,"num_write_kb":556043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":350,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-03T18:13:46.895712+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:13:46.895712+0000","last_peered":"2023-10-03T18:13:46.895712+0000","last_clean":"2023-10-03T18:13:46.895712+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:13:46.895712+0000","last_undegraded":"2023-10-03T18:13:46.895712+0000","last_fullsized":"2023-10-03T18:13:46.895712+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"192'160123","reported_seq":294908,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:52.702204+0000","last_change":"2023-10-03T18:13:51.779304+0000","last_active":"2023-10-03T18:13:52.702204+0000","last_peered":"2023-10-03T18:13:52.702204+0000","last_clean":"2023-10-03T18:13:52.702204+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:13:52.702204+0000","last_undegraded":"2023-10-03T18:13:52.702204+0000","last_fullsized":"2023-10-03T18:13:52.702204+0000","mapping_epoch":12,"log_start":"156'150070","ondisk_log_start":"156'150070","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":77,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":1.599652063,"stat_sum":{"num_bytes":3028763414,"num_objects":5793,"num_object_clones":5095,"num_object_copies":11586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5793,"num_whiteouts":0,"num_read":148794,"num_read_kb":558216,"num_write":141074,"num_write_kb":563409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"3"}]},{"pgid":"2.1","version":"192'151946","reported_seq":274715,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:53.982410+0000","last_change":"2023-10-03T18:13:53.647895+0000","last_active":"2023-10-03T18:13:53.982410+0000","last_peered":"2023-10-03T18:13:53.982410+0000","last_clean":"2023-10-03T18:13:53.982410+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:13:53.982410+0000","last_undegraded":"2023-10-03T18:13:53.982410+0000","last_fullsized":"2023-10-03T18:13:53.982410+0000","mapping_epoch":12,"log_start":"160'141896","ondisk_log_start":"160'141896","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:08:29.548871+0000","scrub_duration":424.08306342600002,"objects_trimmed":42,"snaptrim_duration":1.7672884099999999,"stat_sum":{"num_bytes":2759926281,"num_objects":3185,"num_object_clones":2552,"num_object_copies":6370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":138074,"num_read_kb":507607,"num_write":128692,"num_write_kb":513661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"3"}]},{"pgid":"2.2","version":"192'152339","reported_seq":278739,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-03T18:13:55.145064+0000","last_change":"2023-10-03T18:13:41.081055+0000","last_active":"2023-10-03T18:13:55.145064+0000","last_peered":"2023-10-03T18:13:55.145064+0000","last_clean":"2023-10-03T18:13:55.145064+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:13:55.145064+0000","last_undegraded":"2023-10-03T18:13:55.145064+0000","last_fullsized":"2023-10-03T18:13:55.145064+0000","mapping_epoch":12,"log_start":"157'142318","ondisk_log_start":"157'142318","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:34:17.311310+0000","scrub_duration":370.574046178,"objects_trimmed":60,"snaptrim_duration":4.2043707220000002,"stat_sum":{"num_bytes":2763100983,"num_objects":2997,"num_object_clones":2354,"num_object_copies":5994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2997,"num_whiteouts":0,"num_read":144230,"num_read_kb":515867,"num_write":129810,"num_write_kb":517680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"}]},{"pgid":"2.4","version":"192'146047","reported_seq":269552,"reported_epoch":192,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:13:52.718233+0000","last_change":"2023-10-03T18:13:52.635062+0000","last_active":"2023-10-03T18:13:52.718233+0000","last_peered":"2023-10-03T18:13:52.718233+0000","last_clean":"2023-10-03T18:13:52.718233+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:13:52.718233+0000","last_undegraded":"2023-10-03T18:13:52.718233+0000","last_fullsized":"2023-10-03T18:13:52.718233+0000","mapping_epoch":12,"log_start":"147'135964","ondisk_log_start":"147'135964","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":70,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":2.6055516970000001,"stat_sum":{"num_bytes":2749661871,"num_objects":5057,"num_object_clones":4420,"num_object_copies":10114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5057,"num_whiteouts":0,"num_read":136341,"num_read_kb":505745,"num_write":127992,"num_write_kb":511099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"85","length":"6"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22835934481,"num_objects":42395,"num_object_clones":37131,"num_object_copies":84790,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42395,"num_whiteouts":0,"num_read":1140556,"num_read_kb":4216181,"num_write":1065536,"num_write_kb":4253981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8710090752,"data_stored":63769553442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80458,"ondisk_log_size":80458,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738863,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2593684,"kb_used_data":1634700,"kb_used_omap":969,"kb_used_meta":958006,"kb_avail":91778156,"statfs":{"total":96636764160,"available":93980831744,"internally_reserved":0,"allocated":1673932800,"data_stored":12076862271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":980999022},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,7,3,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.581,"5min":2.669,"15min":2.488},"min":{"1min":0.17899999999999999,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":6.984,"5min":32.173000000000002,"15min":32.173000000000002},"last":15.417},{"interface":"front","average":{"1min":2.3380000000000001,"5min":1.8959999999999999,"15min":2.4390000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":10.994999999999999,"5min":12.731999999999999,"15min":18.041},"last":0.26400000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:13:06 2023","interfaces":[{"interface":"back","average":{"1min":2.823,"5min":2.7509999999999999,"15min":2.5579999999999998},"min":{"1min":0.23499999999999999,"5min":0.21199999999999999,"15min":0.19600000000000001},"max":{"1min":18.768999999999998,"5min":18.768999999999998,"15min":26.074000000000002},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.9590000000000001,"5min":2.7909999999999999,"15min":2.3300000000000001},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":18.751999999999999,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.879}]}]},{"osd":0,"up_from":8,"seq":34359738862,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3436524,"kb_used_data":2623716,"kb_used_omap":2419,"kb_used_meta":810380,"kb_avail":90935316,"statfs":{"total":96636764160,"available":93117763584,"internally_reserved":0,"allocated":2686685184,"data_stored":19795772076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":829829392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":3.6920000000000002,"15min":3.649},"min":{"1min":0.19,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":17.138999999999999,"5min":29.760999999999999,"15min":30.852},"last":5.8529999999999998},{"interface":"front","average":{"1min":3.2799999999999998,"5min":2.7559999999999998,"15min":2.6749999999999998},"min":{"1min":0.22700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":11.291,"5min":27.984999999999999,"15min":27.984999999999999},"last":5.827}]},{"osd":2,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.9359999999999999,"15min":1.907},"min":{"1min":0.249,"5min":0.182,"15min":0.182},"max":{"1min":8.0830000000000002,"5min":12.228999999999999,"15min":14.195},"last":0.253},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.873,"15min":2.0289999999999999},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":7.9050000000000002,"5min":12.880000000000001,"15min":12.880000000000001},"last":5.0570000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738847,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5987780,"kb_used_data":4253752,"kb_used_omap":2305,"kb_used_meta":1731710,"kb_avail":88384060,"statfs":{"total":96636764160,"available":90505277440,"internally_reserved":0,"allocated":4355842048,"data_stored":31914297998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1773271887},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,5,5,12,6,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:13:09 2023","interfaces":[{"interface":"back","average":{"1min":2.1499999999999999,"5min":2.5219999999999998,"15min":2.8410000000000002},"min":{"1min":0.193,"5min":0.17599999999999999,"15min":0.16300000000000001},"max":{"1min":7.1639999999999997,"5min":13.519,"15min":16.997},"last":0.28499999999999998},{"interface":"front","average":{"1min":3.5750000000000002,"5min":2.1560000000000001,"15min":3.194},"min":{"1min":0.26900000000000002,"5min":0.20899999999999999,"15min":0.17899999999999999},"max":{"1min":13.516999999999999,"5min":13.516999999999999,"15min":24.481000000000002},"last":0.438}]},{"osd":2,"last update":"Tue Oct 3 18:13:09 2023","interfaces":[{"interface":"back","average":{"1min":2.258,"5min":2.367,"15min":2.8740000000000001},"min":{"1min":0.29799999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":16.135999999999999,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":3.9169999999999998,"5min":2.3109999999999999,"15min":2.3180000000000001},"min":{"1min":0.307,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":13.581,"5min":13.581,"15min":20.108000000000001},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2684387328,"data_stored":19788754223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4353658880,"data_stored":31904720145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1672044544,"data_stored":12076079074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:13:57.531 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:57.531 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:57.531 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:57.531 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:57.533 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:13:57.533 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:13:59.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:59.865+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:13:59.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:59.881+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:13:59.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:59.881+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:13:59.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:59.881+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:13:59.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:13:59.893+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:00.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:00.343+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:00.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:00.343+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:00.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:00.343+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:00.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:00.382+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:00.397 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:00.384+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:01.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.131+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:01.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.145+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:01.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.158+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:01.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.176+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:01.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.176+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:01.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.176+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:01.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.195+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:01.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.255+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:01.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.440+0000 7fc515f98700 -1 reset not still connected to 0x56221922a1a0 2023-10-03T18:14:01.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.477+0000 7fc515f98700 -1 reset not still connected to 0x56222cfb0340 2023-10-03T18:14:01.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.563+0000 7fc515f98700 -1 reset not still connected to 0x562231b2fe10 2023-10-03T18:14:01.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.563+0000 7fc515f98700 -1 reset not still connected to 0x56220785ba00 2023-10-03T18:14:01.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.563+0000 7fc515f98700 -1 reset not still connected to 0x562218762820 2023-10-03T18:14:01.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.652+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:01.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.687+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:01.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.712+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:01.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.714+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:01.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.794+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:01.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:01.801+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:02.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.368+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:02.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.398+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:02.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.492+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:02.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.511+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:02.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.518+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:02.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.518+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:02.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.518+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:02.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.539+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:02.543 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.541+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:02.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:02.599+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:03.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.848+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:03.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.848+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:03.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.848+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:03.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.860+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:03.868 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.867+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:03.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.886+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:03.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.886+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:03.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.886+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:03.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.934+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:03.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.934+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:03.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:03.934+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:04.296 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:14:04.295+0000 7f0623ebd700 -1 reset not still connected to 0x55873c5eab60 2023-10-03T18:14:05.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:05.244+0000 7fc515f98700 -1 reset not still connected to 0x56221bc292b0 2023-10-03T18:14:07.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.780+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:07.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.780+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:07.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.916+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:07.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.950+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:07.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.950+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:07.952 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.950+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:07.952 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:07.950+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:08.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.132+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:08.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.195+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:08.773 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:14:08.772+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:14:08.788 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:14:08.787+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:14:08.789 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:14:08.787+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:14:08.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.849+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:08.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:08.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.874+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:08.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.874+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:08.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.874+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:08.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.899+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:08.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:08.904+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:10.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.811+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:10.859 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.858+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:10.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.883+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:10.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.883+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:10.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.983+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:10.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.983+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:10.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.983+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:10.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:10.994+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:11.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:11.054+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:15.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.509+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:15.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.579+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:15.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.611+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:15.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.629+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:15.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.731+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:15.745 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.745+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:15.745 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.745+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:15.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.745+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:15.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.745+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:15.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.893+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:15.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.901+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:15.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.955+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:15.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.955+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:15.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.955+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:15.976 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.972+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:15.976 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:15.972+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:16.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.016+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:16.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.016+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:16.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.016+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:16.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.016+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:16.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.097+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:16.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:16.131+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:17.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:14:18.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.041+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:18.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.057+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:18.093 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.092+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:18.093 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.092+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:18.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.115+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:18.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.162+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:18.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:18.187+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:18.291 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:14:18.304 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:14:18.316 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1438,"stamp":"2023-10-03T18:14:16.868024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22861412145,"num_objects":43633,"num_object_clones":38367,"num_object_copies":87266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43633,"num_whiteouts":0,"num_read":1149933,"num_read_kb":4243549,"num_write":1071875,"num_write_kb":4279963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80407,"ondisk_log_size":80407,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12273124,"kb_used_data":8559368,"kb_used_omap":5694,"kb_used_meta":3708033,"kb_avail":270842396,"statfs":{"total":289910292480,"available":277342613504,"internally_reserved":0,"allocated":8764792832,"data_stored":66003551629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":3797026765},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,3,8,6,7,25,9,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10555443,"num_objects":1325,"num_object_clones":1324,"num_object_copies":2650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1325,"num_whiteouts":0,"num_read":3453,"num_read_kb":11343,"num_write":2637,"num_write_kb":10243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.999593"},"pg_stats":[{"pgid":"2.7","version":"201'150819","reported_seq":279190,"reported_epoch":201,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:15.349388+0000","last_change":"2023-10-03T18:14:11.718995+0000","last_active":"2023-10-03T18:14:15.349388+0000","last_peered":"2023-10-03T18:14:15.349388+0000","last_clean":"2023-10-03T18:14:15.349388+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:14:15.349388+0000","last_undegraded":"2023-10-03T18:14:15.349388+0000","last_fullsized":"2023-10-03T18:14:15.349388+0000","mapping_epoch":12,"log_start":"157'140809","ondisk_log_start":"157'140809","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":162,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:07.462891+0000","scrub_duration":505.38444223300002,"objects_trimmed":24,"snaptrim_duration":0.22248545,"stat_sum":{"num_bytes":2849505866,"num_objects":7720,"num_object_clones":7067,"num_object_copies":15440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7720,"num_whiteouts":0,"num_read":140949,"num_read_kb":528247,"num_write":133415,"num_write_kb":532721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"201'139373","reported_seq":258237,"reported_epoch":201,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:15.198354+0000","last_change":"2023-10-03T18:14:11.702964+0000","last_active":"2023-10-03T18:14:15.198354+0000","last_peered":"2023-10-03T18:14:15.198354+0000","last_clean":"2023-10-03T18:14:15.198354+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:14:15.198354+0000","last_undegraded":"2023-10-03T18:14:15.198354+0000","last_fullsized":"2023-10-03T18:14:15.198354+0000","mapping_epoch":12,"log_start":"156'129371","ondisk_log_start":"156'129371","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":82,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.7816650259999998,"stat_sum":{"num_bytes":2631135954,"num_objects":4883,"num_object_clones":4277,"num_object_copies":9766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4883,"num_whiteouts":0,"num_read":131463,"num_read_kb":488477,"num_write":123009,"num_write_kb":491334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"}]},{"pgid":"2.5","version":"201'167873","reported_seq":307773,"reported_epoch":201,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:15.171354+0000","last_change":"2023-10-03T18:14:12.320142+0000","last_active":"2023-10-03T18:14:15.171354+0000","last_peered":"2023-10-03T18:14:15.171354+0000","last_clean":"2023-10-03T18:14:15.171354+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:14:15.171354+0000","last_undegraded":"2023-10-03T18:14:15.171354+0000","last_fullsized":"2023-10-03T18:14:15.171354+0000","mapping_epoch":12,"log_start":"158'157859","ondisk_log_start":"158'157859","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0.40056593099999999,"stat_sum":{"num_bytes":3085374095,"num_objects":6244,"num_object_clones":5534,"num_object_copies":12488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6244,"num_whiteouts":0,"num_read":158635,"num_read_kb":575657,"num_write":144780,"num_write_kb":577555,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"2"}]},{"pgid":"2.0","version":"201'157238","reported_seq":290785,"reported_epoch":201,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:14:14.061414+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:14:14.061414+0000","last_peered":"2023-10-03T18:14:14.061414+0000","last_clean":"2023-10-03T18:14:14.061414+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:14:14.061414+0000","last_undegraded":"2023-10-03T18:14:14.061414+0000","last_fullsized":"2023-10-03T18:14:14.061414+0000","mapping_epoch":12,"log_start":"156'147224","ondisk_log_start":"156'147224","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3401,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 389s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2981274205,"num_objects":7357,"num_object_clones":6673,"num_object_copies":14714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7357,"num_whiteouts":0,"num_read":146678,"num_read_kb":550305,"num_write":139948,"num_write_kb":558913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":359,"reported_epoch":200,"state":"active+clean","last_fresh":"2023-10-03T18:14:06.927998+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:14:06.927998+0000","last_peered":"2023-10-03T18:14:06.927998+0000","last_clean":"2023-10-03T18:14:06.927998+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:14:06.927998+0000","last_undegraded":"2023-10-03T18:14:06.927998+0000","last_fullsized":"2023-10-03T18:14:06.927998+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"201'161522","reported_seq":296944,"reported_epoch":201,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:14.907162+0000","last_change":"2023-10-03T18:14:14.518147+0000","last_active":"2023-10-03T18:14:14.907162+0000","last_peered":"2023-10-03T18:14:14.907162+0000","last_clean":"2023-10-03T18:14:14.907162+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:14:14.907162+0000","last_undegraded":"2023-10-03T18:14:14.907162+0000","last_fullsized":"2023-10-03T18:14:14.907162+0000","mapping_epoch":12,"log_start":"157'151477","ondisk_log_start":"157'151477","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":2.6029033670000001,"stat_sum":{"num_bytes":3032019735,"num_objects":5961,"num_object_clones":5263,"num_object_copies":11922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5961,"num_whiteouts":0,"num_read":149966,"num_read_kb":561964,"num_write":141883,"num_write_kb":566573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"8"}]},{"pgid":"2.1","version":"201'153564","reported_seq":276764,"reported_epoch":201,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:14.083085+0000","last_change":"2023-10-03T18:14:12.749258+0000","last_active":"2023-10-03T18:14:14.083085+0000","last_peered":"2023-10-03T18:14:14.083085+0000","last_clean":"2023-10-03T18:14:14.083085+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:14:14.083085+0000","last_undegraded":"2023-10-03T18:14:14.083085+0000","last_fullsized":"2023-10-03T18:14:14.083085+0000","mapping_epoch":12,"log_start":"170'143503","ondisk_log_start":"170'143503","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:40:58.942253+0000","scrub_duration":424.08306342600002,"objects_trimmed":28,"snaptrim_duration":0.84224189800000004,"stat_sum":{"num_bytes":2762826300,"num_objects":3132,"num_object_clones":2498,"num_object_copies":6264,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":139179,"num_read_kb":510665,"num_write":129398,"num_write_kb":516388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"4"}]},{"pgid":"2.2","version":"201'153695","reported_seq":280616,"reported_epoch":201,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:15.256311+0000","last_change":"2023-10-03T18:14:13.986613+0000","last_active":"2023-10-03T18:14:15.256311+0000","last_peered":"2023-10-03T18:14:15.256311+0000","last_clean":"2023-10-03T18:14:15.256311+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:14:15.256311+0000","last_undegraded":"2023-10-03T18:14:15.256311+0000","last_fullsized":"2023-10-03T18:14:15.256311+0000","mapping_epoch":12,"log_start":"160'143622","ondisk_log_start":"160'143622","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:42:06.484763+0000","scrub_duration":370.574046178,"objects_trimmed":55,"snaptrim_duration":2.099392098,"stat_sum":{"num_bytes":2765898400,"num_objects":3101,"num_object_clones":2461,"num_object_copies":6202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3101,"num_whiteouts":0,"num_read":145504,"num_read_kb":519010,"num_write":130524,"num_write_kb":520408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"5"}]},{"pgid":"2.4","version":"201'147356","reported_seq":271464,"reported_epoch":201,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:15.076872+0000","last_change":"2023-10-03T18:14:14.529410+0000","last_active":"2023-10-03T18:14:15.076872+0000","last_peered":"2023-10-03T18:14:15.076872+0000","last_clean":"2023-10-03T18:14:15.076872+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:14:15.076872+0000","last_undegraded":"2023-10-03T18:14:15.076872+0000","last_fullsized":"2023-10-03T18:14:15.076872+0000","mapping_epoch":12,"log_start":"156'137267","ondisk_log_start":"156'137267","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":86,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":3.8014155540000001,"stat_sum":{"num_bytes":2752787222,"num_objects":5233,"num_object_clones":4594,"num_object_copies":10466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5233,"num_whiteouts":0,"num_read":137453,"num_read_kb":509133,"num_write":128781,"num_write_kb":514177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8a","length":"1"},{"start":"8d","length":"1"},{"start":"96","length":"a"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22860821777,"num_objects":43631,"num_object_clones":38367,"num_object_copies":87262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43631,"num_whiteouts":0,"num_read":1149827,"num_read_kb":4243458,"num_write":1071738,"num_write_kb":4278069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8758374400,"data_stored":66067843618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80308,"ondisk_log_size":80308,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738867,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2639292,"kb_used_data":1643124,"kb_used_omap":969,"kb_used_meta":995190,"kb_avail":91732548,"statfs":{"total":96636764160,"available":93934129152,"internally_reserved":0,"allocated":1682558976,"data_stored":12420681577,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1019075438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,1,1,8,3,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1110000000000002,"5min":2.5419999999999998,"15min":2.4780000000000002},"min":{"1min":0.19700000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":15.417,"5min":17.228999999999999,"15min":32.173000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.9199999999999999,"15min":2.4020000000000001},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":6.9859999999999998,"5min":12.731999999999999,"15min":16.285},"last":0.46500000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":2.4609999999999999,"5min":2.5430000000000001,"15min":2.4689999999999999},"min":{"1min":0.23699999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":7.4000000000000004,"5min":18.768999999999998,"15min":18.768999999999998},"last":3.6509999999999998},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.645,"15min":2.29},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5060000000000002,"5min":18.751999999999999,"15min":19.097000000000001},"last":2.677}]}]},{"osd":0,"up_from":8,"seq":34359738866,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3516904,"kb_used_data":2637472,"kb_used_omap":2419,"kb_used_meta":877004,"kb_avail":90854936,"statfs":{"total":96636764160,"available":93035454464,"internally_reserved":0,"allocated":2700771328,"data_stored":20575688506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":898052368},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,2,0,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":3.7669999999999999,"5min":3.6920000000000002,"15min":3.649},"min":{"1min":0.19,"5min":0.19,"15min":0.17299999999999999},"max":{"1min":17.138999999999999,"5min":29.760999999999999,"15min":30.852},"last":7.1680000000000001},{"interface":"front","average":{"1min":3.2799999999999998,"5min":2.7559999999999998,"15min":2.6749999999999998},"min":{"1min":0.22700000000000001,"5min":0.186,"15min":0.186},"max":{"1min":11.291,"5min":27.984999999999999,"15min":27.984999999999999},"last":6.0419999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:13:19 2023","interfaces":[{"interface":"back","average":{"1min":2.024,"5min":1.9359999999999999,"15min":1.907},"min":{"1min":0.249,"5min":0.182,"15min":0.182},"max":{"1min":8.0830000000000002,"5min":12.228999999999999,"15min":14.195},"last":0.60699999999999998},{"interface":"front","average":{"1min":2.0329999999999999,"5min":1.873,"15min":2.0289999999999999},"min":{"1min":0.23000000000000001,"5min":0.216,"15min":0.20899999999999999},"max":{"1min":7.9050000000000002,"5min":12.880000000000001,"15min":12.880000000000001},"last":4.3860000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738851,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6116928,"kb_used_data":4278772,"kb_used_omap":2305,"kb_used_meta":1835838,"kb_avail":88254912,"statfs":{"total":96636764160,"available":90373029888,"internally_reserved":0,"allocated":4381462528,"data_stored":33007181546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":1879898959},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,3,4,3,4,17,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":2.0499999999999998,"5min":2.3759999999999999,"15min":2.758},"min":{"1min":0.28499999999999998,"5min":0.184,"15min":0.16300000000000001},"max":{"1min":10.529,"5min":10.529,"15min":16.997},"last":0.21199999999999999},{"interface":"front","average":{"1min":2.532,"5min":2.1589999999999998,"15min":3.0089999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":10.571999999999999,"5min":13.516999999999999,"15min":20.716999999999999},"last":1.4159999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":2.085,"15min":2.7040000000000002},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":6.5019999999999998,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.248},{"interface":"front","average":{"1min":2.8370000000000002,"5min":2.4279999999999999,"15min":2.3700000000000001},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":11.458,"5min":13.581,"15min":20.108000000000001},"last":1.4930000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2698366976,"data_stored":20589501793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4379426816,"data_stored":33055679761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1680580608,"data_stored":12422662064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:14:18.318 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:18.318 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:18.319 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:18.319 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:18.319 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:18.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:14:19.004 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:14:19.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:14:19.775 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:14:19.793 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:14:20.511 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:14:20.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:14:21.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.052+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:21.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.102+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:21.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.102+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:21.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.102+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:21.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.246+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:21.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.246+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:21.255 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:14:21.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:14:21.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.326+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:21.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.326+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:21.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.326+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:21.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.326+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:21.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:21.326+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:21.964 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:14:21.984 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:14:25.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.309+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:25.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.447+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:25.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.469+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:25.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.469+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:25.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.489+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:25.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.489+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:25.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.489+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:25.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.542+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:25.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:25.555+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:33.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.014+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:33.015 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.014+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:33.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.017+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:33.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.048+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:33.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.048+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:33.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.069+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:33.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.069+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:33.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.116+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:33.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.162+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:33.205 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:33.204+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:34.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:34.671+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:34.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:34.671+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:34.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:34.733+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:34.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:34.733+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:34.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:34.733+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:35.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:35.374+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:35.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:35.374+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:35.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:35.374+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:35.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:35.374+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:35.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:35.374+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:36.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.179+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:36.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.182+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:36.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.233+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:36.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.285+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:36.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.285+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:36.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.412+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:36.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.412+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:36.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.468+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:36.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.769+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:36.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.769+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:36.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.877+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:36.887 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:36.887+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:39.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.344+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:39.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.353+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:39.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.420+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:39.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.420+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:39.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.420+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:39.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.427+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:39.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.522+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:39.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:39.527+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:41.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:14:42.727 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:14:42.729 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:14:42.747 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1454,"stamp":"2023-10-03T18:14:41.908002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22887049009,"num_objects":44440,"num_object_clones":39174,"num_object_copies":88880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44440,"num_whiteouts":0,"num_read":1157954,"num_read_kb":4270905,"num_write":1078101,"num_write_kb":4304867,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80628,"ondisk_log_size":80628,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12680904,"kb_used_data":8609252,"kb_used_omap":5694,"kb_used_meta":4065921,"kb_avail":270434616,"statfs":{"total":289910292480,"available":276925046784,"internally_reserved":0,"allocated":8815874048,"data_stored":67643620925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":4163504077},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,3,4,3,8,15,18],"upper_bound":256},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8630272,"num_objects":17,"num_object_clones":17,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":2975,"num_read_kb":9756,"num_write":2096,"num_write_kb":8384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.017644"},"pg_stats":[{"pgid":"2.7","version":"204'152001","reported_seq":280875,"reported_epoch":204,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:37.249353+0000","last_change":"2023-10-03T18:14:36.878054+0000","last_active":"2023-10-03T18:14:37.249353+0000","last_peered":"2023-10-03T18:14:37.249353+0000","last_clean":"2023-10-03T18:14:37.249353+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:14:37.249353+0000","last_undegraded":"2023-10-03T18:14:37.249353+0000","last_fullsized":"2023-10-03T18:14:37.249353+0000","mapping_epoch":12,"log_start":"159'141915","ondisk_log_start":"159'141915","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":158,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:07.462891+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":4.8962184899999999,"stat_sum":{"num_bytes":2852487754,"num_objects":7900,"num_object_clones":7247,"num_object_copies":15800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7900,"num_whiteouts":0,"num_read":141844,"num_read_kb":531399,"num_write":134144,"num_write_kb":535637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"97","length":"1"}]},{"pgid":"2.6","version":"204'140546","reported_seq":259896,"reported_epoch":204,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:37.166813+0000","last_change":"2023-10-03T18:14:35.721431+0000","last_active":"2023-10-03T18:14:37.166813+0000","last_peered":"2023-10-03T18:14:37.166813+0000","last_clean":"2023-10-03T18:14:37.166813+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:14:37.166813+0000","last_undegraded":"2023-10-03T18:14:37.166813+0000","last_fullsized":"2023-10-03T18:14:37.166813+0000","mapping_epoch":12,"log_start":"157'130473","ondisk_log_start":"157'130473","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":79,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":4.769349257,"stat_sum":{"num_bytes":2633978578,"num_objects":5000,"num_object_clones":4394,"num_object_copies":10000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5000,"num_whiteouts":0,"num_read":132316,"num_read_kb":491419,"num_write":123702,"num_write_kb":494106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"1"}]},{"pgid":"2.5","version":"204'169208","reported_seq":309695,"reported_epoch":204,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:37.194259+0000","last_change":"2023-10-03T18:14:33.959177+0000","last_active":"2023-10-03T18:14:37.194259+0000","last_peered":"2023-10-03T18:14:37.194259+0000","last_clean":"2023-10-03T18:14:37.194259+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:14:37.194259+0000","last_undegraded":"2023-10-03T18:14:37.194259+0000","last_fullsized":"2023-10-03T18:14:37.194259+0000","mapping_epoch":12,"log_start":"162'159162","ondisk_log_start":"162'159162","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.976431676,"stat_sum":{"num_bytes":3088810639,"num_objects":6432,"num_object_clones":5722,"num_object_copies":12864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6432,"num_whiteouts":0,"num_read":159737,"num_read_kb":579209,"num_write":145609,"num_write_kb":580871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"5"}]},{"pgid":"2.0","version":"204'158623","reported_seq":292908,"reported_epoch":204,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:14:39.331491+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:14:39.331491+0000","last_peered":"2023-10-03T18:14:39.331491+0000","last_clean":"2023-10-03T18:14:39.331491+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:14:39.331491+0000","last_undegraded":"2023-10-03T18:14:39.331491+0000","last_fullsized":"2023-10-03T18:14:39.331491+0000","mapping_epoch":12,"log_start":"158'148527","ondisk_log_start":"158'148527","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3648,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 414s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2984788573,"num_objects":7650,"num_object_clones":6966,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7650,"num_whiteouts":0,"num_read":147805,"num_read_kb":554377,"num_write":140806,"num_write_kb":562345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":362,"reported_epoch":203,"state":"active+clean","last_fresh":"2023-10-03T18:14:31.990567+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:14:31.990567+0000","last_peered":"2023-10-03T18:14:31.990567+0000","last_clean":"2023-10-03T18:14:31.990567+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:14:31.990567+0000","last_undegraded":"2023-10-03T18:14:31.990567+0000","last_fullsized":"2023-10-03T18:14:31.990567+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"204'162831","reported_seq":298779,"reported_epoch":204,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:37.084346+0000","last_change":"2023-10-03T18:14:36.884561+0000","last_active":"2023-10-03T18:14:37.084346+0000","last_peered":"2023-10-03T18:14:37.084346+0000","last_clean":"2023-10-03T18:14:37.084346+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:14:37.084346+0000","last_undegraded":"2023-10-03T18:14:37.084346+0000","last_fullsized":"2023-10-03T18:14:37.084346+0000","mapping_epoch":12,"log_start":"161'152784","ondisk_log_start":"161'152784","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":89,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0539381590000001,"stat_sum":{"num_bytes":3035243287,"num_objects":6144,"num_object_clones":5446,"num_object_copies":12288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6144,"num_whiteouts":0,"num_read":150918,"num_read_kb":565332,"num_write":142670,"num_write_kb":569721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"1"}]},{"pgid":"2.1","version":"204'155450","reported_seq":279141,"reported_epoch":204,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:39.331636+0000","last_change":"2023-10-03T18:14:38.913899+0000","last_active":"2023-10-03T18:14:39.331636+0000","last_peered":"2023-10-03T18:14:39.331636+0000","last_clean":"2023-10-03T18:14:39.331636+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:14:39.331636+0000","last_undegraded":"2023-10-03T18:14:39.331636+0000","last_fullsized":"2023-10-03T18:14:39.331636+0000","mapping_epoch":12,"log_start":"173'145413","ondisk_log_start":"173'145413","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:40:58.942253+0000","scrub_duration":424.08306342600002,"objects_trimmed":46,"snaptrim_duration":1.950482394,"stat_sum":{"num_bytes":2766131772,"num_objects":3003,"num_object_clones":2369,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":140234,"num_read_kb":514219,"num_write":130201,"num_write_kb":519600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"}]},{"pgid":"2.2","version":"204'155574","reported_seq":283042,"reported_epoch":204,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:40.499091+0000","last_change":"2023-10-03T18:14:38.614603+0000","last_active":"2023-10-03T18:14:40.499091+0000","last_peered":"2023-10-03T18:14:40.499091+0000","last_clean":"2023-10-03T18:14:40.499091+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:14:40.499091+0000","last_undegraded":"2023-10-03T18:14:40.499091+0000","last_fullsized":"2023-10-03T18:14:40.499091+0000","mapping_epoch":12,"log_start":"170'145530","ondisk_log_start":"170'145530","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:42:06.484763+0000","scrub_duration":370.574046178,"objects_trimmed":55,"snaptrim_duration":1.6818501910000001,"stat_sum":{"num_bytes":2769318560,"num_objects":2953,"num_object_clones":2313,"num_object_copies":5906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2953,"num_whiteouts":0,"num_read":146641,"num_read_kb":522625,"num_write":131349,"num_write_kb":523708,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"}]},{"pgid":"2.4","version":"204'148571","reported_seq":273194,"reported_epoch":204,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:37.146606+0000","last_change":"2023-10-03T18:14:35.702053+0000","last_active":"2023-10-03T18:14:37.146606+0000","last_peered":"2023-10-03T18:14:37.146606+0000","last_clean":"2023-10-03T18:14:37.146606+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:14:37.146606+0000","last_undegraded":"2023-10-03T18:14:37.146606+0000","last_fullsized":"2023-10-03T18:14:37.146606+0000","mapping_epoch":12,"log_start":"157'138471","ondisk_log_start":"157'138471","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":82,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.7201190029999998,"stat_sum":{"num_bytes":2755699478,"num_objects":5356,"num_object_clones":4717,"num_object_copies":10712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5356,"num_whiteouts":0,"num_read":138353,"num_read_kb":512234,"num_write":129483,"num_write_kb":516985,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"97","length":"9"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22886458641,"num_objects":44438,"num_object_clones":39174,"num_object_copies":88876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44438,"num_whiteouts":0,"num_read":1157848,"num_read_kb":4270814,"num_write":1077964,"num_write_kb":4302973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8810221568,"data_stored":67637586466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80529,"ondisk_log_size":80529,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2734256,"kb_used_data":1653412,"kb_used_omap":969,"kb_used_meta":1079862,"kb_avail":91637584,"statfs":{"total":96636764160,"available":93836886016,"internally_reserved":0,"allocated":1693093888,"data_stored":12746852521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1105779566},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,1,2,1,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1110000000000002,"5min":2.5419999999999998,"15min":2.4780000000000002},"min":{"1min":0.19700000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":15.417,"5min":17.228999999999999,"15min":32.173000000000002},"last":1.867},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.9199999999999999,"15min":2.4020000000000001},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":6.9859999999999998,"5min":12.731999999999999,"15min":16.285},"last":0.25}]},{"osd":1,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":2.4609999999999999,"5min":2.5430000000000001,"15min":2.4689999999999999},"min":{"1min":0.23699999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":7.4000000000000004,"5min":18.768999999999998,"15min":18.768999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.645,"15min":2.29},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5060000000000002,"5min":18.751999999999999,"15min":19.097000000000001},"last":1.51}]}]},{"osd":0,"up_from":8,"seq":34359738871,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3655828,"kb_used_data":2654216,"kb_used_omap":2419,"kb_used_meta":999180,"kb_avail":90716012,"statfs":{"total":96636764160,"available":92893196288,"internally_reserved":0,"allocated":2717917184,"data_stored":21074927738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1023160592},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":3.04,"5min":3.9159999999999999,"15min":3.5609999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":9.4149999999999991,"5min":29.760999999999999,"15min":30.852},"last":2.1230000000000002},{"interface":"front","average":{"1min":3.5830000000000002,"5min":3.1760000000000002,"15min":2.7530000000000001},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.186},"max":{"1min":16.082000000000001,"5min":27.984999999999999,"15min":27.984999999999999},"last":2.157}]},{"osd":2,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.861,"15min":1.9370000000000001},"min":{"1min":0.24299999999999999,"5min":0.182,"15min":0.182},"max":{"1min":13.209,"5min":13.209,"15min":14.195},"last":0.28199999999999997},{"interface":"front","average":{"1min":3.3050000000000002,"5min":2.1949999999999998,"15min":2.0699999999999998},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.214},"max":{"1min":16.033999999999999,"5min":16.033999999999999,"15min":16.033999999999999},"last":6.9850000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6290820,"kb_used_data":4301624,"kb_used_omap":2305,"kb_used_meta":1986878,"kb_avail":88081020,"statfs":{"total":96636764160,"available":90194964480,"internally_reserved":0,"allocated":4404862976,"data_stored":33821840666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":2034563919},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,1,1,6,14,14],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":2.0499999999999998,"5min":2.3759999999999999,"15min":2.758},"min":{"1min":0.28499999999999998,"5min":0.184,"15min":0.16300000000000001},"max":{"1min":10.529,"5min":10.529,"15min":16.997},"last":2.1539999999999999},{"interface":"front","average":{"1min":2.532,"5min":2.1589999999999998,"15min":3.0089999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":10.571999999999999,"5min":13.516999999999999,"15min":20.716999999999999},"last":2.0699999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":2.085,"15min":2.7040000000000002},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":6.5019999999999998,"5min":21.326000000000001,"15min":24.579000000000001},"last":13.948},{"interface":"front","average":{"1min":2.8370000000000002,"5min":2.4279999999999999,"15min":2.3700000000000001},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":11.458,"5min":13.581,"15min":20.108000000000001},"last":11.111000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2715742208,"data_stored":21076053345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4403412992,"data_stored":33816392977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1691066368,"data_stored":12745140144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:14:42.749 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:42.749 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:42.749 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:42.750 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:42.750 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:14:42.750 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:14:47.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:47.898+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:47.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:47.898+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:47.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:47.898+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:48.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.056+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:48.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.056+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:48.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.070+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:48.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.077+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:48.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.077+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:48.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.105+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:48.140 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.140+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:48.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.156+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:48.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.166+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:48.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.187+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:48.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.197+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:48.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.211+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:48.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.223+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:48.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.246+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:48.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.281+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:48.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.281+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:48.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.303+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:48.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.306+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:48.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.306+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:48.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.623+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:48.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.623+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:48.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.689+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:48.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.689+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:48.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:48.689+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:49.466 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.460+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:49.466 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.460+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:49.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.473+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:49.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.473+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:49.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.473+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:49.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.510+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:49.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:49.510+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:52.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.229+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:52.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.229+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:52.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.229+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:52.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.235+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:52.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.235+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:52.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.301+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:52.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.301+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:52.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.322+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:52.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.388+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:52.405 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:52.404+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:53.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.360+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:53.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.360+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:53.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.360+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:53.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.366+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:53.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.366+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:53.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.396+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:53.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.396+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:53.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.396+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:53.397 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.396+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:53.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.402+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:53.409 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:53.402+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:54.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:54.173+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:54.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:54.173+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:54.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:54.189+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:54.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:54.189+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:54.279 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:54.276+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:55.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.351+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:55.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.392+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:55.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.392+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:55.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.393+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:55.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.427+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:55.428 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.427+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:55.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.637+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:14:55.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.739+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:14:55.742 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.742+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:55.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.766+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:55.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.779+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:55.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.779+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:55.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.830+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:14:55.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.831+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:14:55.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.839+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:55.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.868+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:14:55.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:55.887+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:14:56.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.769+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:14:56.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.778+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:14:56.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.781+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:14:56.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.789+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:14:56.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.816+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:14:56.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:14:56.816+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:00.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:00.890+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:00.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:00.890+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:00.923 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:00.921+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:00.923 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:00.921+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:00.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:00.924+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:01.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.088+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:01.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.088+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:01.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.489+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:01.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.639+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:01.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.639+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:01.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.639+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:01.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.639+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:01.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.746+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:01.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:01.746+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:02.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.322+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:02.453 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.451+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:02.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.451+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:02.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.497+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:02.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.521+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:02.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.521+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:02.545 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.544+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:02.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.561+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:02.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:02.711+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:02.751 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:15:03.298 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:03.294+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:15:03.319 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:03.315+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:15:03.319 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:03.315+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:15:03.320 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:03.315+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:15:03.520 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:15:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:15:03.550 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1470,"stamp":"2023-10-03T18:15:02.994773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22917932849,"num_objects":44358,"num_object_clones":39092,"num_object_copies":88716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44358,"num_whiteouts":0,"num_read":1166004,"num_read_kb":4302469,"num_write":1085551,"num_write_kb":4334667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80456,"ondisk_log_size":80456,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12890460,"kb_used_data":8668288,"kb_used_omap":5694,"kb_used_meta":4216449,"kb_avail":270225060,"statfs":{"total":289910292480,"available":276710461440,"internally_reserved":0,"allocated":8876326912,"data_stored":67570035933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5830707,"internal_metadata":4317644749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,8,19,10,14,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8204288,"num_objects":-81,"num_object_clones":-81,"num_object_copies":-162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-81,"num_whiteouts":0,"num_read":2231,"num_read_kb":8460,"num_write":1982,"num_write_kb":7928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.079172"},"pg_stats":[{"pgid":"2.7","version":"207'153264","reported_seq":283067,"reported_epoch":207,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:58.230849+0000","last_change":"2023-10-03T18:14:54.918659+0000","last_active":"2023-10-03T18:14:58.230849+0000","last_peered":"2023-10-03T18:14:58.230849+0000","last_clean":"2023-10-03T18:14:58.230849+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:14:58.230849+0000","last_undegraded":"2023-10-03T18:14:58.230849+0000","last_fullsized":"2023-10-03T18:14:58.230849+0000","mapping_epoch":12,"log_start":"167'143217","ondisk_log_start":"167'143217","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":155,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:53:07.462891+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":2.978524095,"stat_sum":{"num_bytes":2856395338,"num_objects":7869,"num_object_clones":7216,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7869,"num_whiteouts":0,"num_read":142876,"num_read_kb":535503,"num_write":135089,"num_write_kb":539417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"1"}]},{"pgid":"2.6","version":"207'141738","reported_seq":261911,"reported_epoch":207,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:58.227059+0000","last_change":"2023-10-03T18:14:56.431499+0000","last_active":"2023-10-03T18:14:58.227059+0000","last_peered":"2023-10-03T18:14:58.227059+0000","last_clean":"2023-10-03T18:14:58.227059+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:14:58.227059+0000","last_undegraded":"2023-10-03T18:14:58.227059+0000","last_fullsized":"2023-10-03T18:14:58.227059+0000","mapping_epoch":12,"log_start":"161'131677","ondisk_log_start":"161'131677","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.4600942999999997,"stat_sum":{"num_bytes":2637492946,"num_objects":4972,"num_object_clones":4366,"num_object_copies":9944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":133257,"num_read_kb":495155,"num_write":124556,"num_write_kb":497522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"6"}]},{"pgid":"2.5","version":"207'170683","reported_seq":312039,"reported_epoch":207,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:58.217294+0000","last_change":"2023-10-03T18:14:56.432379+0000","last_active":"2023-10-03T18:14:58.217294+0000","last_peered":"2023-10-03T18:14:58.217294+0000","last_clean":"2023-10-03T18:14:58.217294+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:14:58.217294+0000","last_undegraded":"2023-10-03T18:14:58.217294+0000","last_fullsized":"2023-10-03T18:14:58.217294+0000","mapping_epoch":12,"log_start":"172'160669","ondisk_log_start":"172'160669","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":1.022469128,"stat_sum":{"num_bytes":3093131919,"num_objects":6383,"num_object_clones":5673,"num_object_copies":12766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6383,"num_whiteouts":0,"num_read":160740,"num_read_kb":583185,"num_write":146651,"num_write_kb":585039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"8"}]},{"pgid":"2.0","version":"207'159829","reported_seq":295139,"reported_epoch":207,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:14:59.593011+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:14:59.593011+0000","last_peered":"2023-10-03T18:14:59.593011+0000","last_clean":"2023-10-03T18:14:59.593011+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:14:59.593011+0000","last_undegraded":"2023-10-03T18:14:59.593011+0000","last_fullsized":"2023-10-03T18:14:59.593011+0000","mapping_epoch":12,"log_start":"165'149728","ondisk_log_start":"165'149728","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3812,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 435s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2988683869,"num_objects":7751,"num_object_clones":7067,"num_object_copies":15502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7751,"num_whiteouts":0,"num_read":148895,"num_read_kb":558705,"num_write":141752,"num_write_kb":566129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":365,"reported_epoch":206,"state":"active+clean","last_fresh":"2023-10-03T18:14:51.950921+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:14:51.950921+0000","last_peered":"2023-10-03T18:14:51.950921+0000","last_clean":"2023-10-03T18:14:51.950921+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:14:51.950921+0000","last_undegraded":"2023-10-03T18:14:51.950921+0000","last_fullsized":"2023-10-03T18:14:51.950921+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"207'164195","reported_seq":301027,"reported_epoch":207,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:14:58.154219+0000","last_change":"2023-10-03T18:14:56.703698+0000","last_active":"2023-10-03T18:14:58.154219+0000","last_peered":"2023-10-03T18:14:58.154219+0000","last_clean":"2023-10-03T18:14:58.154219+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:14:58.154219+0000","last_undegraded":"2023-10-03T18:14:58.154219+0000","last_fullsized":"2023-10-03T18:14:58.154219+0000","mapping_epoch":12,"log_start":"171'154187","ondisk_log_start":"171'154187","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":85,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.7517104640000003,"stat_sum":{"num_bytes":3039380247,"num_objects":6096,"num_object_clones":5398,"num_object_copies":12192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":151923,"num_read_kb":569336,"num_write":143661,"num_write_kb":573685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"3"}]},{"pgid":"2.1","version":"207'156589","reported_seq":281121,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-10-03T18:14:59.598498+0000","last_change":"2023-10-03T18:14:43.283531+0000","last_active":"2023-10-03T18:14:59.598498+0000","last_peered":"2023-10-03T18:14:59.598498+0000","last_clean":"2023-10-03T18:14:59.598498+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:14:59.598498+0000","last_undegraded":"2023-10-03T18:14:59.598498+0000","last_fullsized":"2023-10-03T18:14:59.598498+0000","mapping_epoch":12,"log_start":"175'146517","ondisk_log_start":"175'146517","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:40:58.942253+0000","scrub_duration":424.08306342600002,"objects_trimmed":21,"snaptrim_duration":1.3363381510000001,"stat_sum":{"num_bytes":2769969724,"num_objects":3006,"num_object_clones":2372,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":141155,"num_read_kb":517891,"num_write":131112,"num_write_kb":523244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"}]},{"pgid":"2.2","version":"207'156578","reported_seq":284984,"reported_epoch":207,"state":"active+clean","last_fresh":"2023-10-03T18:15:00.538585+0000","last_change":"2023-10-03T18:14:42.305169+0000","last_active":"2023-10-03T18:15:00.538585+0000","last_peered":"2023-10-03T18:15:00.538585+0000","last_clean":"2023-10-03T18:15:00.538585+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:15:00.538585+0000","last_undegraded":"2023-10-03T18:15:00.538585+0000","last_fullsized":"2023-10-03T18:15:00.538585+0000","mapping_epoch":12,"log_start":"172'146531","ondisk_log_start":"172'146531","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:42:06.484763+0000","scrub_duration":370.574046178,"objects_trimmed":5,"snaptrim_duration":0.38465891400000002,"stat_sum":{"num_bytes":2772894368,"num_objects":2993,"num_object_clones":2353,"num_object_copies":5986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2993,"num_whiteouts":0,"num_read":147726,"num_read_kb":526537,"num_write":132220,"num_write_kb":527192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"}]},{"pgid":"2.4","version":"207'149884","reported_seq":275344,"reported_epoch":207,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:14:58.197081+0000","last_change":"2023-10-03T18:14:56.703762+0000","last_active":"2023-10-03T18:14:58.197081+0000","last_peered":"2023-10-03T18:14:58.197081+0000","last_clean":"2023-10-03T18:14:58.197081+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:14:58.197081+0000","last_undegraded":"2023-10-03T18:14:58.197081+0000","last_fullsized":"2023-10-03T18:14:58.197081+0000","mapping_epoch":12,"log_start":"166'139877","ondisk_log_start":"166'139877","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":78,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":2.82144079,"stat_sum":{"num_bytes":2759394070,"num_objects":5286,"num_object_clones":4647,"num_object_copies":10572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5286,"num_whiteouts":0,"num_read":139326,"num_read_kb":516066,"num_write":130373,"num_write_kb":520545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"6"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22917342481,"num_objects":44356,"num_object_clones":39092,"num_object_copies":88712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44356,"num_whiteouts":0,"num_read":1165898,"num_read_kb":4302378,"num_write":1085414,"num_write_kb":4332773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8871268352,"data_stored":67559741986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58201,"internal_metadata":0},"log_size":80357,"ondisk_log_size":80357,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2755208,"kb_used_data":1664640,"kb_used_omap":969,"kb_used_meta":1089590,"kb_avail":91616632,"statfs":{"total":96636764160,"available":93815431168,"internally_reserved":0,"allocated":1704591360,"data_stored":12831392217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1115741038},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,0,1,2,1,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":3.1110000000000002,"5min":2.5419999999999998,"15min":2.4780000000000002},"min":{"1min":0.19700000000000001,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":15.417,"5min":17.228999999999999,"15min":32.173000000000002},"last":0.189},{"interface":"front","average":{"1min":2.0899999999999999,"5min":1.9199999999999999,"15min":2.4020000000000001},"min":{"1min":0.218,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":6.9859999999999998,"5min":12.731999999999999,"15min":16.285},"last":0.26300000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:14:07 2023","interfaces":[{"interface":"back","average":{"1min":2.4609999999999999,"5min":2.5430000000000001,"15min":2.4689999999999999},"min":{"1min":0.23699999999999999,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":7.4000000000000004,"5min":18.768999999999998,"15min":18.768999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.645,"15min":2.29},"min":{"1min":0.23400000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5060000000000002,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.42199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738875,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3728880,"kb_used_data":2672552,"kb_used_omap":2419,"kb_used_meta":1053900,"kb_avail":90642960,"statfs":{"total":96636764160,"available":92818391040,"internally_reserved":0,"allocated":2736693248,"data_stored":20939368874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1079193872},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,10,0,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":3.04,"5min":3.9159999999999999,"15min":3.5609999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":9.4149999999999991,"5min":29.760999999999999,"15min":30.852},"last":1.7050000000000001},{"interface":"front","average":{"1min":3.5830000000000002,"5min":3.1760000000000002,"15min":2.7530000000000001},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.186},"max":{"1min":16.082000000000001,"5min":27.984999999999999,"15min":27.984999999999999},"last":1.629}]},{"osd":2,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.861,"15min":1.9370000000000001},"min":{"1min":0.24299999999999999,"5min":0.182,"15min":0.182},"max":{"1min":13.209,"5min":13.209,"15min":14.195},"last":4.774},{"interface":"front","average":{"1min":3.3050000000000002,"5min":2.1949999999999998,"15min":2.0699999999999998},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.214},"max":{"1min":16.033999999999999,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.224}]}]},{"osd":1,"up_from":8,"seq":34359738859,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6406372,"kb_used_data":4331096,"kb_used_omap":2305,"kb_used_meta":2072958,"kb_avail":87965468,"statfs":{"total":96636764160,"available":90076639232,"internally_reserved":0,"allocated":4435042304,"data_stored":33799274842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2360497,"internal_metadata":2122709839},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,7,8,8,10],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":2.0499999999999998,"5min":2.3759999999999999,"15min":2.758},"min":{"1min":0.28499999999999998,"5min":0.184,"15min":0.16300000000000001},"max":{"1min":10.529,"5min":10.529,"15min":16.997},"last":3.782},{"interface":"front","average":{"1min":2.532,"5min":2.1589999999999998,"15min":3.0089999999999999},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":10.571999999999999,"5min":13.516999999999999,"15min":20.716999999999999},"last":3.8109999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:14:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":2.085,"15min":2.7040000000000002},"min":{"1min":0.222,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":6.5019999999999998,"5min":21.326000000000001,"15min":24.579000000000001},"last":3.9169999999999998},{"interface":"front","average":{"1min":2.8370000000000002,"5min":2.4279999999999999,"15min":2.3700000000000001},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.17299999999999999},"max":{"1min":11.458,"5min":13.581,"15min":20.108000000000001},"last":0.90100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2734493696,"data_stored":20938136929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4434063360,"data_stored":33788286225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28896,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1702711296,"data_stored":12833318832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:15:03.552 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:03.552 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:03.552 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:03.553 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:03.553 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:03.553 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:15:04.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.375+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:04.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.375+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:04.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.410+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:04.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.428+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:04.438 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.436+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:04.463 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:04.457+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:04.595 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:04.594+0000 7f9f21f20700 -1 reset not still connected to 0x5632f083dc70 2023-10-03T18:15:04.595 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:04.594+0000 7f9f21f20700 -1 reset not still connected to 0x5632f13a15f0 2023-10-03T18:15:13.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.240+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:13.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.241+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:13.298 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.297+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:13.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.360+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:13.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.612+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:13.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.689+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:13.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.783+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:13.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.783+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:13.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.783+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:13.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.831+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:13.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:13.959+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:18.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.167+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:18.209 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.207+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:18.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.222+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:18.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.222+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:18.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.388+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:18.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.479+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:18.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.626+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:18.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.626+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:18.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.626+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:18.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.698+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:18.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:18.717+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:20.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.702+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:20.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.887+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:20.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.941+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:20.970 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.970+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:20.970 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.970+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:20.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:20.989+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:21.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:21.020+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:21.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:21.020+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:21.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:21.039+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:21.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:21.066+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:21.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:21.090+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:23.553 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:15:24.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:15:24.297 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:15:24.316 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1486,"stamp":"2023-10-03T18:15:22.920680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22940530481,"num_objects":46222,"num_object_clones":40956,"num_object_copies":92444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46222,"num_whiteouts":0,"num_read":1174559,"num_read_kb":4327321,"num_write":1091167,"num_write_kb":4356411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80542,"ondisk_log_size":80542,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":12795160,"kb_used_data":8711616,"kb_used_omap":5697,"kb_used_meta":4077822,"kb_avail":270320360,"statfs":{"total":289910292480,"available":276808048640,"internally_reserved":0,"allocated":8920694784,"data_stored":70917225191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5834541,"internal_metadata":4175689939},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,8,11,22,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10752000,"num_objects":1078,"num_object_clones":1078,"num_object_copies":2156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1078,"num_whiteouts":0,"num_read":2671,"num_read_kb":10640,"num_write":2573,"num_write_kb":10292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006360"},"pg_stats":[{"pgid":"2.7","version":"215'154575","reported_seq":284810,"reported_epoch":215,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:20.664069+0000","last_change":"2023-10-03T18:15:17.008831+0000","last_active":"2023-10-03T18:15:20.664069+0000","last_peered":"2023-10-03T18:15:20.664069+0000","last_clean":"2023-10-03T18:15:20.664069+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:15:20.664069+0000","last_undegraded":"2023-10-03T18:15:20.664069+0000","last_fullsized":"2023-10-03T18:15:20.664069+0000","mapping_epoch":12,"log_start":"172'144520","ondisk_log_start":"172'144520","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":172,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:24:39.696354+0000","scrub_duration":505.38444223300002,"objects_trimmed":29,"snaptrim_duration":5.0403488440000004,"stat_sum":{"num_bytes":2859434469,"num_objects":8149,"num_object_clones":7498,"num_object_copies":16298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8149,"num_whiteouts":0,"num_read":143901,"num_read_kb":538665,"num_write":135845,"num_write_kb":542361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9d","length":"1"}]},{"pgid":"2.6","version":"215'142931","reported_seq":263565,"reported_epoch":215,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:15:20.696207+0000","last_change":"2023-10-03T18:15:19.237709+0000","last_active":"2023-10-03T18:15:20.696207+0000","last_peered":"2023-10-03T18:15:20.696207+0000","last_clean":"2023-10-03T18:15:20.696207+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:15:20.696207+0000","last_undegraded":"2023-10-03T18:15:20.696207+0000","last_fullsized":"2023-10-03T18:15:20.696207+0000","mapping_epoch":12,"log_start":"170'132884","ondisk_log_start":"170'132884","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":91,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":43,"snaptrim_duration":2.1488761630000002,"stat_sum":{"num_bytes":2640253601,"num_objects":5203,"num_object_clones":4598,"num_object_copies":10406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5203,"num_whiteouts":0,"num_read":134161,"num_read_kb":498054,"num_write":125232,"num_write_kb":500172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"}]},{"pgid":"2.5","version":"215'172066","reported_seq":313990,"reported_epoch":215,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:15:20.390874+0000","last_change":"2023-10-03T18:15:19.589993+0000","last_active":"2023-10-03T18:15:20.390874+0000","last_peered":"2023-10-03T18:15:20.390874+0000","last_clean":"2023-10-03T18:15:20.390874+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:15:20.390874+0000","last_undegraded":"2023-10-03T18:15:20.390874+0000","last_fullsized":"2023-10-03T18:15:20.390874+0000","mapping_epoch":12,"log_start":"174'161976","ondisk_log_start":"174'161976","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":109,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.8298406699999998,"stat_sum":{"num_bytes":3096253020,"num_objects":6690,"num_object_clones":5981,"num_object_copies":13380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6690,"num_whiteouts":0,"num_read":162131,"num_read_kb":586853,"num_write":147454,"num_write_kb":588110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"1"}]},{"pgid":"2.0","version":"215'160998","reported_seq":296706,"reported_epoch":215,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:15:19.673011+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:15:19.673011+0000","last_peered":"2023-10-03T18:15:19.673011+0000","last_clean":"2023-10-03T18:15:19.673011+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:15:19.673011+0000","last_undegraded":"2023-10-03T18:15:19.673011+0000","last_fullsized":"2023-10-03T18:15:19.673011+0000","mapping_epoch":12,"log_start":"172'150934","ondisk_log_start":"172'150934","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":3954,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 455s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2991444675,"num_objects":8116,"num_object_clones":7430,"num_object_copies":16232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8116,"num_whiteouts":0,"num_read":149836,"num_read_kb":561547,"num_write":142429,"num_write_kb":568773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":373,"reported_epoch":214,"state":"active+clean","last_fresh":"2023-10-03T18:15:06.943080+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:15:06.943080+0000","last_peered":"2023-10-03T18:15:06.943080+0000","last_clean":"2023-10-03T18:15:06.943080+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:15:06.943080+0000","last_undegraded":"2023-10-03T18:15:06.943080+0000","last_fullsized":"2023-10-03T18:15:06.943080+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"215'165582","reported_seq":302850,"reported_epoch":215,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:20.206310+0000","last_change":"2023-10-03T18:15:19.580130+0000","last_active":"2023-10-03T18:15:20.206310+0000","last_peered":"2023-10-03T18:15:20.206310+0000","last_clean":"2023-10-03T18:15:20.206310+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:15:20.206310+0000","last_undegraded":"2023-10-03T18:15:20.206310+0000","last_fullsized":"2023-10-03T18:15:20.206310+0000","mapping_epoch":12,"log_start":"173'155492","ondisk_log_start":"173'155492","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":102,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":7.573458037,"stat_sum":{"num_bytes":3042620282,"num_objects":6389,"num_object_clones":5689,"num_object_copies":12778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6389,"num_whiteouts":0,"num_read":152981,"num_read_kb":572652,"num_write":144441,"num_write_kb":576727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"}]},{"pgid":"2.1","version":"215'158049","reported_seq":282929,"reported_epoch":215,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:15:19.641094+0000","last_change":"2023-10-03T18:15:18.845156+0000","last_active":"2023-10-03T18:15:19.641094+0000","last_peered":"2023-10-03T18:15:19.641094+0000","last_clean":"2023-10-03T18:15:19.641094+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:15:19.641094+0000","last_undegraded":"2023-10-03T18:15:19.641094+0000","last_fullsized":"2023-10-03T18:15:19.641094+0000","mapping_epoch":12,"log_start":"182'148022","ondisk_log_start":"182'148022","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:45:15.117478+0000","scrub_duration":424.08306342600002,"objects_trimmed":114,"snaptrim_duration":6.8849223029999997,"stat_sum":{"num_bytes":2772521482,"num_objects":3089,"num_object_clones":2456,"num_object_copies":6178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3089,"num_whiteouts":0,"num_read":142216,"num_read_kb":520762,"num_write":131739,"num_write_kb":525654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"8"}]},{"pgid":"2.2","version":"215'158064","reported_seq":286842,"reported_epoch":215,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:15:20.752217+0000","last_change":"2023-10-03T18:15:19.817077+0000","last_active":"2023-10-03T18:15:20.752217+0000","last_peered":"2023-10-03T18:15:20.752217+0000","last_clean":"2023-10-03T18:15:20.752217+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:15:20.752217+0000","last_undegraded":"2023-10-03T18:15:20.752217+0000","last_fullsized":"2023-10-03T18:15:20.752217+0000","mapping_epoch":12,"log_start":"175'148047","ondisk_log_start":"175'148047","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:52:08.250556+0000","scrub_duration":370.574046178,"objects_trimmed":119,"snaptrim_duration":7.8675588579999998,"stat_sum":{"num_bytes":2775270099,"num_objects":3055,"num_object_clones":2414,"num_object_copies":6110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3055,"num_whiteouts":0,"num_read":148852,"num_read_kb":529501,"num_write":132821,"num_write_kb":529477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"9"}]},{"pgid":"2.4","version":"215'151136","reported_seq":277078,"reported_epoch":215,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:20.121152+0000","last_change":"2023-10-03T18:15:19.234976+0000","last_active":"2023-10-03T18:15:20.121152+0000","last_peered":"2023-10-03T18:15:20.121152+0000","last_clean":"2023-10-03T18:15:20.121152+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:15:20.121152+0000","last_undegraded":"2023-10-03T18:15:20.121152+0000","last_fullsized":"2023-10-03T18:15:20.121152+0000","mapping_epoch":12,"log_start":"172'141083","ondisk_log_start":"172'141083","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":94,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":33,"snaptrim_duration":7.2493675790000003,"stat_sum":{"num_bytes":2762142485,"num_objects":5529,"num_object_clones":4890,"num_object_copies":11058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5529,"num_whiteouts":0,"num_read":140375,"num_read_kb":519196,"num_write":131069,"num_write_kb":523243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9d","length":"3"},{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22939940113,"num_objects":46220,"num_object_clones":40956,"num_object_copies":92440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46220,"num_whiteouts":0,"num_read":1174453,"num_read_kb":4327230,"num_write":1091030,"num_write_kb":4354517,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8914837504,"data_stored":71005243938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62035,"internal_metadata":0},"log_size":80443,"ondisk_log_size":80443,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738880,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2847704,"kb_used_data":1672528,"kb_used_omap":969,"kb_used_meta":1174198,"kb_avail":91524136,"statfs":{"total":96636764160,"available":93720715264,"internally_reserved":0,"allocated":1712668672,"data_stored":13477406494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1202379630},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":4.1109999999999998,"5min":3,"15min":2.532},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":18.321000000000002,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.73099999999999998},{"interface":"front","average":{"1min":3.2400000000000002,"5min":2.23,"15min":2.4209999999999998},"min":{"1min":0.25,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":8.9469999999999992,"5min":10.994999999999999,"15min":16.285},"last":0.71199999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":3.2930000000000001,"5min":2.6499999999999999,"15min":2.5600000000000001},"min":{"1min":0.40300000000000002,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":12.673999999999999,"5min":18.768999999999998,"15min":18.768999999999998},"last":1.843},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.5449999999999999,"15min":2.379},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":8.7349999999999994,"5min":18.751999999999999,"15min":19.097000000000001},"last":1.177}]}]},{"osd":0,"up_from":8,"seq":34359738879,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3848240,"kb_used_data":2684456,"kb_used_omap":2419,"kb_used_meta":1161356,"kb_avail":90523600,"statfs":{"total":96636764160,"available":92696166400,"internally_reserved":0,"allocated":2748882944,"data_stored":21995515329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1189228816},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":3.04,"5min":3.9159999999999999,"15min":3.5609999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.17299999999999999},"max":{"1min":9.4149999999999991,"5min":29.760999999999999,"15min":30.852},"last":31.474},{"interface":"front","average":{"1min":3.5830000000000002,"5min":3.1760000000000002,"15min":2.7530000000000001},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.186},"max":{"1min":16.082000000000001,"5min":27.984999999999999,"15min":27.984999999999999},"last":30.273}]},{"osd":2,"last update":"Tue Oct 3 18:14:22 2023","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.861,"15min":1.9370000000000001},"min":{"1min":0.24299999999999999,"5min":0.182,"15min":0.182},"max":{"1min":13.209,"5min":13.209,"15min":14.195},"last":0.23599999999999999},{"interface":"front","average":{"1min":3.3050000000000002,"5min":2.1949999999999998,"15min":2.0699999999999998},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.214},"max":{"1min":16.033999999999999,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.26400000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738863,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6099216,"kb_used_data":4354632,"kb_used_omap":2308,"kb_used_meta":1742267,"kb_avail":88272624,"statfs":{"total":96636764160,"available":90391166976,"internally_reserved":0,"allocated":4459143168,"data_stored":35444303368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2364331,"internal_metadata":1784081493},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,5,1,7,10,16,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":2.488,"15min":2.673},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":10.492000000000001,"5min":10.529,"15min":16.997},"last":0.20100000000000001},{"interface":"front","average":{"1min":2.645,"5min":2.4079999999999999,"15min":2.9780000000000002},"min":{"1min":0.35999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.2199999999999998,"5min":13.516999999999999,"15min":20.716999999999999},"last":0.33400000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1230000000000002,"5min":2.0979999999999999,"15min":2.6299999999999999},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.948,"5min":21.326000000000001,"15min":24.579000000000001},"last":2.6680000000000001},{"interface":"front","average":{"1min":3.0009999999999999,"5min":2.669,"15min":2.4769999999999999},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.17299999999999999},"max":{"1min":11.111000000000001,"5min":13.581,"15min":20.108000000000001},"last":0.52800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2746679296,"data_stored":22013127882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4457594880,"data_stored":35515933969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32730,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1710563328,"data_stored":13476182087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:15:24.318 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:24.318 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:24.318 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:24.318 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:24.319 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:24.319 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:15:24.781 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:24.780+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:15:24.807 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:24.806+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:15:24.807 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:24.807+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:15:24.810 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:24.810+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:15:29.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.866+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:29.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.866+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:29.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.869+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:29.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.869+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:29.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.869+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:29.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.971+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:29.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.971+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:29.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:29.971+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:31.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.535+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:31.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.535+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:31.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.535+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:31.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.607+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:31.625 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.619+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:31.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.651+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:31.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.664+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:31.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.664+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:31.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.681+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:31.737 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.735+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:31.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.735+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:31.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.977+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:31.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.977+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:31.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.977+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:31.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.977+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:31.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:31.977+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:32.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.140+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:32.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.157+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:32.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.157+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:32.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.157+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:32.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.241+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:32.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.755+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:32.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.755+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:32.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.782+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:32.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.835+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:32.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:32.899+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:33.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.064+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:33.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.076+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:33.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.082+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:33.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.082+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:33.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.112+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:33.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.112+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:33.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.124+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:33.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.149+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:33.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:33.161+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:35.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.608+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:35.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.687+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:35.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.705+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:35.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.726+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:35.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.739+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:35.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.739+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:35.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.774+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:35.793 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.793+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:35.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:35.800+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:36.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.057+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:36.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.059+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:36.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.059+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:36.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.159+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:36.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.159+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:36.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.159+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:36.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.159+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:36.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.195+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:36.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:36.195+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:37.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.318+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:37.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.501+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:37.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.554+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:37.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.554+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:37.565 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.564+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:37.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.616+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:37.635 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.631+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:37.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.670+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:37.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.674+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:37.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.674+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:37.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.701+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:37.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.722+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:37.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.722+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:37.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.722+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:37.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.722+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:37.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.722+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:37.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.769+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:37.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.779+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:37.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:37.789+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:39.516 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:39.515+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:15:39.517 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:39.517+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:15:39.577 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:39.574+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:15:39.587 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:39.586+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:15:40.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.388+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:40.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.435+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:40.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.443+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:40.478 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.468+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:40.478 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.468+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:40.563 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:40.563+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:41.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.296+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:41.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.328+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:41.359 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.356+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:41.359 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.356+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:41.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.362+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:41.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.370+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:41.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.370+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:41.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.370+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:41.440 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.440+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:41.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.456+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:41.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:41.456+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:44.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:15:45.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:15:45.089 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:15:45.098 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1500,"stamp":"2023-10-03T18:15:44.941697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22967772977,"num_objects":46680,"num_object_clones":41414,"num_object_copies":93360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46680,"num_whiteouts":0,"num_read":1182680,"num_read_kb":4355096,"num_write":1097583,"num_write_kb":4382075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80418,"ondisk_log_size":80418,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13018308,"kb_used_data":8761700,"kb_used_omap":5697,"kb_used_meta":4250878,"kb_avail":270097212,"statfs":{"total":289910292480,"available":276579545088,"internally_reserved":0,"allocated":8971980800,"data_stored":71902104679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5834541,"internal_metadata":4352899283},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,3,3,1,5,14,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12697600,"num_objects":71,"num_object_clones":71,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":4432,"num_read_kb":13508,"num_write":2972,"num_write_kb":11888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.018588"},"pg_stats":[{"pgid":"2.7","version":"219'155780","reported_seq":286539,"reported_epoch":219,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:42.008928+0000","last_change":"2023-10-03T18:15:42.008928+0000","last_active":"2023-10-03T18:15:42.008928+0000","last_peered":"2023-10-03T18:15:42.008928+0000","last_clean":"2023-10-03T18:15:42.008928+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:15:42.008928+0000","last_undegraded":"2023-10-03T18:15:42.008928+0000","last_fullsized":"2023-10-03T18:15:42.008928+0000","mapping_epoch":12,"log_start":"175'145725","ondisk_log_start":"175'145725","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":168,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:24:39.696354+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.017673662999999999,"stat_sum":{"num_bytes":2862420453,"num_objects":8211,"num_object_clones":7560,"num_object_copies":16422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8211,"num_whiteouts":0,"num_read":144759,"num_read_kb":541657,"num_write":136565,"num_write_kb":545241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a9","length":"1"}]},{"pgid":"2.6","version":"219'144054","reported_seq":265253,"reported_epoch":219,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:15:41.996026+0000","last_change":"2023-10-03T18:15:41.785322+0000","last_active":"2023-10-03T18:15:41.996026+0000","last_peered":"2023-10-03T18:15:41.996026+0000","last_clean":"2023-10-03T18:15:41.996026+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:15:41.996026+0000","last_undegraded":"2023-10-03T18:15:41.996026+0000","last_fullsized":"2023-10-03T18:15:41.996026+0000","mapping_epoch":12,"log_start":"173'133990","ondisk_log_start":"173'133990","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":87,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.7987853620000001,"stat_sum":{"num_bytes":2643403425,"num_objects":5307,"num_object_clones":4702,"num_object_copies":10614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5307,"num_whiteouts":0,"num_read":135031,"num_read_kb":501063,"num_write":125974,"num_write_kb":503140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a9","length":"3"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"4"}]},{"pgid":"2.5","version":"218'173382","reported_seq":315946,"reported_epoch":219,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:41.993792+0000","last_change":"2023-10-03T18:15:40.754887+0000","last_active":"2023-10-03T18:15:41.993792+0000","last_peered":"2023-10-03T18:15:41.993792+0000","last_clean":"2023-10-03T18:15:41.993792+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:15:41.993792+0000","last_undegraded":"2023-10-03T18:15:41.993792+0000","last_fullsized":"2023-10-03T18:15:41.993792+0000","mapping_epoch":12,"log_start":"183'163380","ondisk_log_start":"183'163380","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":105,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.754978242,"stat_sum":{"num_bytes":3099652700,"num_objects":6785,"num_object_clones":6076,"num_object_copies":13570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6785,"num_whiteouts":0,"num_read":163221,"num_read_kb":590345,"num_write":148261,"num_write_kb":591338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a9","length":"3"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"3"}]},{"pgid":"2.0","version":"219'162442","reported_seq":299000,"reported_epoch":219,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:15:44.859900+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:15:44.859900+0000","last_peered":"2023-10-03T18:15:44.859900+0000","last_clean":"2023-10-03T18:15:44.859900+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:15:44.859900+0000","last_undegraded":"2023-10-03T18:15:44.859900+0000","last_fullsized":"2023-10-03T18:15:44.859900+0000","mapping_epoch":12,"log_start":"175'152439","ondisk_log_start":"175'152439","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":4162,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 480s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2995733187,"num_objects":8367,"num_object_clones":7681,"num_object_copies":16734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8367,"num_whiteouts":0,"num_read":151016,"num_read_kb":565831,"num_write":143423,"num_write_kb":572749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a9","length":"3"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":376,"reported_epoch":217,"state":"active+clean","last_fresh":"2023-10-03T18:15:31.985660+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:15:31.985660+0000","last_peered":"2023-10-03T18:15:31.985660+0000","last_clean":"2023-10-03T18:15:31.985660+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:15:31.985660+0000","last_undegraded":"2023-10-03T18:15:31.985660+0000","last_fullsized":"2023-10-03T18:15:31.985660+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"218'166876","reported_seq":304777,"reported_epoch":219,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:41.991533+0000","last_change":"2023-10-03T18:15:40.306034+0000","last_active":"2023-10-03T18:15:41.991533+0000","last_peered":"2023-10-03T18:15:41.991533+0000","last_clean":"2023-10-03T18:15:41.991533+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:15:41.991533+0000","last_undegraded":"2023-10-03T18:15:41.991533+0000","last_fullsized":"2023-10-03T18:15:41.991533+0000","mapping_epoch":12,"log_start":"177'156796","ondisk_log_start":"177'156796","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":99,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.3022530359999998,"stat_sum":{"num_bytes":3046175610,"num_objects":6498,"num_object_clones":5798,"num_object_copies":12996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6498,"num_whiteouts":0,"num_read":153957,"num_read_kb":576124,"num_write":145288,"num_write_kb":580115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a9","length":"3"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"1"}]},{"pgid":"2.1","version":"219'159929","reported_seq":285490,"reported_epoch":219,"state":"active+clean","last_fresh":"2023-10-03T18:15:44.860115+0000","last_change":"2023-10-03T18:15:41.086700+0000","last_active":"2023-10-03T18:15:44.860115+0000","last_peered":"2023-10-03T18:15:44.860115+0000","last_clean":"2023-10-03T18:15:44.860115+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:15:44.860115+0000","last_undegraded":"2023-10-03T18:15:44.860115+0000","last_fullsized":"2023-10-03T18:15:44.860115+0000","mapping_epoch":12,"log_start":"188'149836","ondisk_log_start":"188'149836","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:45:15.117478+0000","scrub_duration":424.08306342600002,"objects_trimmed":67,"snaptrim_duration":4.1122121610000004,"stat_sum":{"num_bytes":2776506890,"num_objects":2955,"num_object_clones":2322,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2955,"num_whiteouts":0,"num_read":143398,"num_read_kb":524840,"num_write":132661,"num_write_kb":529342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a9","length":"3"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"}]},{"pgid":"2.2","version":"218'159565","reported_seq":288866,"reported_epoch":218,"state":"active+clean","last_fresh":"2023-10-03T18:15:40.841436+0000","last_change":"2023-10-03T18:15:39.617606+0000","last_active":"2023-10-03T18:15:40.841436+0000","last_peered":"2023-10-03T18:15:40.841436+0000","last_clean":"2023-10-03T18:15:40.841436+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:15:40.841436+0000","last_undegraded":"2023-10-03T18:15:40.841436+0000","last_fullsized":"2023-10-03T18:15:40.841436+0000","mapping_epoch":12,"log_start":"187'149552","ondisk_log_start":"187'149552","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:52:08.250556+0000","scrub_duration":370.574046178,"objects_trimmed":47,"snaptrim_duration":2.5635766580000001,"stat_sum":{"num_bytes":2778133203,"num_objects":2929,"num_object_clones":2288,"num_object_copies":5858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2929,"num_whiteouts":0,"num_read":149874,"num_read_kb":532672,"num_write":133475,"num_write_kb":532093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"}]},{"pgid":"2.4","version":"218'152298","reported_seq":278855,"reported_epoch":218,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:15:41.966761+0000","last_change":"2023-10-03T18:15:41.778856+0000","last_active":"2023-10-03T18:15:41.966761+0000","last_peered":"2023-10-03T18:15:41.966761+0000","last_clean":"2023-10-03T18:15:41.966761+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:15:41.966761+0000","last_undegraded":"2023-10-03T18:15:41.966761+0000","last_fullsized":"2023-10-03T18:15:41.966761+0000","mapping_epoch":12,"log_start":"174'142289","ondisk_log_start":"174'142289","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":90,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":4.7390175570000004,"stat_sum":{"num_bytes":2765157141,"num_objects":5626,"num_object_clones":4987,"num_object_copies":11252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":141318,"num_read_kb":522473,"num_write":131799,"num_write_kb":526163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"a8","length":"4"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22967182609,"num_objects":46678,"num_object_clones":41414,"num_object_copies":93356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46678,"num_whiteouts":0,"num_read":1182574,"num_read_kb":4355005,"num_write":1097446,"num_write_kb":4380181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8965787648,"data_stored":71916706338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62035,"internal_metadata":0},"log_size":80319,"ondisk_log_size":80319,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738885,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2873176,"kb_used_data":1684112,"kb_used_omap":969,"kb_used_meta":1188086,"kb_avail":91498664,"statfs":{"total":96636764160,"available":93694631936,"internally_reserved":0,"allocated":1724530688,"data_stored":13700965502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1216600942},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3,0,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":4.1109999999999998,"5min":3,"15min":2.532},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":18.321000000000002,"5min":18.321000000000002,"15min":32.173000000000002},"last":5.2350000000000003},{"interface":"front","average":{"1min":3.2400000000000002,"5min":2.23,"15min":2.4209999999999998},"min":{"1min":0.25,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":8.9469999999999992,"5min":10.994999999999999,"15min":16.285},"last":0.38200000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":3.2930000000000001,"5min":2.6499999999999999,"15min":2.5600000000000001},"min":{"1min":0.40300000000000002,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":12.673999999999999,"5min":18.768999999999998,"15min":18.768999999999998},"last":4.7329999999999997},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.5449999999999999,"15min":2.379},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":8.7349999999999994,"5min":18.751999999999999,"15min":19.097000000000001},"last":3.694}]}]},{"osd":0,"up_from":8,"seq":34359738883,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3935448,"kb_used_data":2698576,"kb_used_omap":2419,"kb_used_meta":1234444,"kb_avail":90436392,"statfs":{"total":96636764160,"available":92606865408,"internally_reserved":0,"allocated":2763341824,"data_stored":22266484689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1264070928},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2640000000000002,"5min":3.895,"15min":3.6309999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":31.474,"5min":31.474,"15min":31.474},"last":0.88},{"interface":"front","average":{"1min":4.6580000000000004,"5min":3.605,"15min":2.9700000000000002},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":30.273,"5min":30.273,"15min":30.273},"last":6.9009999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.839,"15min":1.9690000000000001},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":5.4669999999999996,"5min":13.209,"15min":14.195},"last":0.31},{"interface":"front","average":{"1min":1.8520000000000001,"5min":2.0819999999999999,"15min":2.1120000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.214},"max":{"1min":7.0060000000000002,"5min":16.033999999999999,"15min":16.033999999999999},"last":1.302}]}]},{"osd":1,"up_from":8,"seq":34359738867,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6209684,"kb_used_data":4379012,"kb_used_omap":2308,"kb_used_meta":1828347,"kb_avail":88162156,"statfs":{"total":96636764160,"available":90278047744,"internally_reserved":0,"allocated":4484108288,"data_stored":35934654488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2364331,"internal_metadata":1872227413},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1,4,13,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":2.488,"15min":2.673},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":10.492000000000001,"5min":10.529,"15min":16.997},"last":3.831},{"interface":"front","average":{"1min":2.645,"5min":2.4079999999999999,"15min":2.9780000000000002},"min":{"1min":0.35999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.2199999999999998,"5min":13.516999999999999,"15min":20.716999999999999},"last":6.585}]},{"osd":2,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1230000000000002,"5min":2.0979999999999999,"15min":2.6299999999999999},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.948,"5min":21.326000000000001,"15min":24.579000000000001},"last":3.7719999999999998},{"interface":"front","average":{"1min":3.0009999999999999,"5min":2.669,"15min":2.4769999999999999},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.17299999999999999},"max":{"1min":11.111000000000001,"5min":13.581,"15min":20.108000000000001},"last":6.6230000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2760699904,"data_stored":22269529290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4482531328,"data_stored":35941975313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32730,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1722556416,"data_stored":13705201735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:15:45.100 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:45.100 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:45.100 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:45.100 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:45.100 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:15:45.101 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:15:49.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.492+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:49.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.492+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:49.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.503+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:49.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.514+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:49.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.514+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:49.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.529+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:49.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.529+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:49.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:49.530+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:51.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.035+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:51.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.202+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:51.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.202+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:51.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.220+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:51.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.220+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:51.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.280+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:51.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.280+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:51.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.280+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:51.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.280+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:51.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.280+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:51.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.990+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:52.004 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.994+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:52.004 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:51.994+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:52.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.005+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:52.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.005+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:52.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.005+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:52.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.005+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:52.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.009+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:52.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.091+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:52.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.600+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:15:52.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:15:52.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:52.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:52.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:52.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:52.601+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:56.052 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:56.051+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:15:56.052 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:56.051+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:15:56.052 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:15:56.051+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:15:56.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.098+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:56.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.102+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:56.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.152+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:56.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.222+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:56.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.223+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:56.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.230+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:56.261 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.261+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:15:56.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.728+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:15:56.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.761+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:15:56.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.777+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:15:56.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.777+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:15:56.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.797+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:15:56.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.797+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:15:56.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.797+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:15:56.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.797+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:15:56.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:15:56.797+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:00.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.054+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:00.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.068+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:00.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.068+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:00.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.068+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:00.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.083+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:00.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.125+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:00.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.125+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:00.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.125+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:00.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.188+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:00.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:00.209+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:01.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:01.228+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:01.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:01.242+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:01.323 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:01.321+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:01.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:01.351+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:01.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:01.415+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:02.677 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:16:02.674+0000 7f0623ebd700 -1 reset not still connected to 0x55873b754dd0 2023-10-03T18:16:03.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:03.682+0000 7fc515f98700 -1 reset not still connected to 0x56220af3a270 2023-10-03T18:16:03.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:03.720+0000 7fc515f98700 -1 reset not still connected to 0x56221843d040 2023-10-03T18:16:05.102 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:16:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:16:05.972 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:16:05.991 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1517,"stamp":"2023-10-03T18:16:05.017952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":22999795458,"num_objects":46648,"num_object_clones":41383,"num_object_copies":93296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46648,"num_whiteouts":0,"num_read":1191798,"num_read_kb":4387093,"num_write":1105033,"num_write_kb":4411632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80429,"ondisk_log_size":80429,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13292004,"kb_used_data":8821632,"kb_used_omap":5697,"kb_used_meta":4464638,"kb_avail":269823516,"statfs":{"total":289910292480,"available":276299280384,"internally_reserved":0,"allocated":9033351168,"data_stored":71973011627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5834541,"internal_metadata":4571789523},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,8,7,5,12,15,8,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5382097,"num_objects":10,"num_object_clones":11,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":2547,"num_read_kb":6257,"num_write":1309,"num_write_kb":4993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036882"},"pg_stats":[{"pgid":"2.7","version":"224'157191","reported_seq":288876,"reported_epoch":224,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:03.484924+0000","last_change":"2023-10-03T18:16:01.683150+0000","last_active":"2023-10-03T18:16:03.484924+0000","last_peered":"2023-10-03T18:16:03.484924+0000","last_clean":"2023-10-03T18:16:03.484924+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:16:03.484924+0000","last_undegraded":"2023-10-03T18:16:03.484924+0000","last_fullsized":"2023-10-03T18:16:03.484924+0000","mapping_epoch":12,"log_start":"187'147127","ondisk_log_start":"187'147127","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":174,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:57:54.264928+0000","scrub_duration":505.38444223300002,"objects_trimmed":4,"snaptrim_duration":2.5140600700000002,"stat_sum":{"num_bytes":2866848179,"num_objects":8174,"num_object_clones":7524,"num_object_copies":16348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8174,"num_whiteouts":0,"num_read":145971,"num_read_kb":545836,"num_write":137598,"num_write_kb":549340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"af","length":"1"}]},{"pgid":"2.6","version":"224'145339","reported_seq":267535,"reported_epoch":224,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:03.384583+0000","last_change":"2023-10-03T18:16:03.376721+0000","last_active":"2023-10-03T18:16:03.384583+0000","last_peered":"2023-10-03T18:16:03.384583+0000","last_clean":"2023-10-03T18:16:03.384583+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:16:03.384583+0000","last_undegraded":"2023-10-03T18:16:03.384583+0000","last_fullsized":"2023-10-03T18:16:03.384583+0000","mapping_epoch":12,"log_start":"178'135292","ondisk_log_start":"178'135292","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0.29037165599999998,"stat_sum":{"num_bytes":2647270100,"num_objects":5241,"num_object_clones":4635,"num_object_copies":10482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":136258,"num_read_kb":505284,"num_write":126864,"num_write_kb":506661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"af","length":"2"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"9"}]},{"pgid":"2.5","version":"224'174890","reported_seq":318514,"reported_epoch":224,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:03.459152+0000","last_change":"2023-10-03T18:16:01.654942+0000","last_active":"2023-10-03T18:16:03.459152+0000","last_peered":"2023-10-03T18:16:03.459152+0000","last_clean":"2023-10-03T18:16:03.459152+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:16:03.459152+0000","last_undegraded":"2023-10-03T18:16:03.459152+0000","last_fullsized":"2023-10-03T18:16:03.459152+0000","mapping_epoch":12,"log_start":"188'164884","ondisk_log_start":"188'164884","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":111,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.63882819700000004,"stat_sum":{"num_bytes":3104371343,"num_objects":6759,"num_object_clones":6049,"num_object_copies":13518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6759,"num_whiteouts":0,"num_read":164697,"num_read_kb":595021,"num_write":149366,"num_write_kb":595681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"af","length":"2"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"1"}]},{"pgid":"2.0","version":"221'163350","reported_seq":300688,"reported_epoch":221,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:16:00.006384+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:16:00.006384+0000","last_peered":"2023-10-03T18:16:00.006384+0000","last_clean":"2023-10-03T18:16:00.006384+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:16:00.006384+0000","last_undegraded":"2023-10-03T18:16:00.006384+0000","last_fullsized":"2023-10-03T18:16:00.006384+0000","mapping_epoch":12,"log_start":"183'153340","ondisk_log_start":"183'153340","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":4280,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 495s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":2998952643,"num_objects":8410,"num_object_clones":7724,"num_object_copies":16820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8410,"num_whiteouts":0,"num_read":151826,"num_read_kb":569055,"num_write":144174,"num_write_kb":575753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"1"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":383,"reported_epoch":223,"state":"active+clean","last_fresh":"2023-10-03T18:16:02.069916+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:16:02.069916+0000","last_peered":"2023-10-03T18:16:02.069916+0000","last_clean":"2023-10-03T18:16:02.069916+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:16:02.069916+0000","last_undegraded":"2023-10-03T18:16:02.069916+0000","last_fullsized":"2023-10-03T18:16:02.069916+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"224'168321","reported_seq":307283,"reported_epoch":224,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:03.431960+0000","last_change":"2023-10-03T18:15:59.545762+0000","last_active":"2023-10-03T18:16:03.431960+0000","last_peered":"2023-10-03T18:16:03.431960+0000","last_clean":"2023-10-03T18:16:03.431960+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:16:03.431960+0000","last_undegraded":"2023-10-03T18:16:03.431960+0000","last_fullsized":"2023-10-03T18:16:03.431960+0000","mapping_epoch":12,"log_start":"188'158300","ondisk_log_start":"188'158300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":105,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.5646041020000001,"stat_sum":{"num_bytes":3050918729,"num_objects":6466,"num_object_clones":5767,"num_object_copies":12932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6466,"num_whiteouts":0,"num_read":155306,"num_read_kb":580830,"num_write":146356,"num_write_kb":584333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"af","length":"2"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"5"}]},{"pgid":"2.1","version":"221'160671","reported_seq":286930,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-03T18:16:00.028651+0000","last_change":"2023-10-03T18:15:41.086700+0000","last_active":"2023-10-03T18:16:00.028651+0000","last_peered":"2023-10-03T18:16:00.028651+0000","last_clean":"2023-10-03T18:16:00.028651+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:16:00.028651+0000","last_undegraded":"2023-10-03T18:16:00.028651+0000","last_fullsized":"2023-10-03T18:16:00.028651+0000","mapping_epoch":12,"log_start":"189'150636","ondisk_log_start":"189'150636","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:45:15.117478+0000","scrub_duration":424.08306342600002,"objects_trimmed":67,"snaptrim_duration":4.1122121610000004,"stat_sum":{"num_bytes":2779435530,"num_objects":2998,"num_object_clones":2365,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":144127,"num_read_kb":527740,"num_write":133354,"num_write_kb":532114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ab","length":"1"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"}]},{"pgid":"2.2","version":"221'160543","reported_seq":290776,"reported_epoch":221,"state":"active+clean","last_fresh":"2023-10-03T18:16:00.962018+0000","last_change":"2023-10-03T18:15:39.617606+0000","last_active":"2023-10-03T18:16:00.962018+0000","last_peered":"2023-10-03T18:16:00.962018+0000","last_clean":"2023-10-03T18:16:00.962018+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:16:00.962018+0000","last_undegraded":"2023-10-03T18:16:00.962018+0000","last_fullsized":"2023-10-03T18:16:00.962018+0000","mapping_epoch":12,"log_start":"188'150453","ondisk_log_start":"188'150453","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:52:08.250556+0000","scrub_duration":370.574046178,"objects_trimmed":47,"snaptrim_duration":2.5635766580000001,"stat_sum":{"num_bytes":2781958867,"num_objects":3004,"num_object_clones":2363,"num_object_copies":6008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3004,"num_whiteouts":0,"num_read":150958,"num_read_kb":536584,"num_write":134372,"num_write_kb":535681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ab","length":"1"},{"start":"ae","length":"3"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"}]},{"pgid":"2.4","version":"224'153648","reported_seq":281172,"reported_epoch":224,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:03.419236+0000","last_change":"2023-10-03T18:16:01.038002+0000","last_active":"2023-10-03T18:16:03.419236+0000","last_peered":"2023-10-03T18:16:03.419236+0000","last_clean":"2023-10-03T18:16:03.419236+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:16:03.419236+0000","last_undegraded":"2023-10-03T18:16:03.419236+0000","last_fullsized":"2023-10-03T18:16:03.419236+0000","mapping_epoch":12,"log_start":"186'143591","ondisk_log_start":"186'143591","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":96,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.0412594620000002,"stat_sum":{"num_bytes":2769449699,"num_objects":5594,"num_object_clones":4956,"num_object_copies":11188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5594,"num_whiteouts":0,"num_read":142549,"num_read_kb":526652,"num_write":132812,"num_write_kb":530175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"af","length":"2"},{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":22999205090,"num_objects":46646,"num_object_clones":41383,"num_object_copies":93292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46646,"num_whiteouts":0,"num_read":1191692,"num_read_kb":4387002,"num_write":1104896,"num_write_kb":4409738,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9027006464,"data_stored":72013367944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62035,"internal_metadata":0},"log_size":80330,"ondisk_log_size":80330,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738888,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2909664,"kb_used_data":1692884,"kb_used_omap":969,"kb_used_meta":1215798,"kb_avail":91462176,"statfs":{"total":96636764160,"available":93657268224,"internally_reserved":0,"allocated":1733513216,"data_stored":13746764574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1244978030},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,0,0,3,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":4.1109999999999998,"5min":3,"15min":2.532},"min":{"1min":0.189,"5min":0.16600000000000001,"15min":0.16600000000000001},"max":{"1min":18.321000000000002,"5min":18.321000000000002,"15min":32.173000000000002},"last":1.6259999999999999},{"interface":"front","average":{"1min":3.2400000000000002,"5min":2.23,"15min":2.4209999999999998},"min":{"1min":0.25,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":8.9469999999999992,"5min":10.994999999999999,"15min":16.285},"last":1.462}]},{"osd":1,"last update":"Tue Oct 3 18:15:07 2023","interfaces":[{"interface":"back","average":{"1min":3.2930000000000001,"5min":2.6499999999999999,"15min":2.5600000000000001},"min":{"1min":0.40300000000000002,"5min":0.21199999999999999,"15min":0.20399999999999999},"max":{"1min":12.673999999999999,"5min":18.768999999999998,"15min":18.768999999999998},"last":2.2109999999999999},{"interface":"front","average":{"1min":2.4380000000000002,"5min":2.5449999999999999,"15min":2.379},"min":{"1min":0.23599999999999999,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":8.7349999999999994,"5min":18.751999999999999,"15min":19.097000000000001},"last":2.1850000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738887,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4018108,"kb_used_data":2717364,"kb_used_omap":2419,"kb_used_meta":1298316,"kb_avail":90353732,"statfs":{"total":96636764160,"available":92522221568,"internally_reserved":0,"allocated":2782580736,"data_stored":22196982209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1329475856},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2640000000000002,"5min":3.895,"15min":3.6309999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":31.474,"5min":31.474,"15min":31.474},"last":0.36599999999999999},{"interface":"front","average":{"1min":4.6580000000000004,"5min":3.605,"15min":2.9700000000000002},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":30.273,"5min":30.273,"15min":30.273},"last":0.40699999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.839,"15min":1.9690000000000001},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":5.4669999999999996,"5min":13.209,"15min":14.195},"last":3.6259999999999999},{"interface":"front","average":{"1min":1.8520000000000001,"5min":2.0819999999999999,"15min":2.1120000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.214},"max":{"1min":7.0060000000000002,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.32700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6364232,"kb_used_data":4411384,"kb_used_omap":2308,"kb_used_meta":1950523,"kb_avail":88007608,"statfs":{"total":96636764160,"available":90119790592,"internally_reserved":0,"allocated":4517257216,"data_stored":36029264844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2364331,"internal_metadata":1997335637},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,7,4,8,11,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.2229999999999999,"5min":2.488,"15min":2.673},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":10.492000000000001,"5min":10.529,"15min":16.997},"last":0.26700000000000002},{"interface":"front","average":{"1min":2.645,"5min":2.4079999999999999,"15min":2.9780000000000002},"min":{"1min":0.35999999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":6.2199999999999998,"5min":13.516999999999999,"15min":20.716999999999999},"last":1.4159999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:15:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1230000000000002,"5min":2.0979999999999999,"15min":2.6299999999999999},"min":{"1min":0.223,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":13.948,"5min":21.326000000000001,"15min":24.579000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":3.0009999999999999,"5min":2.669,"15min":2.4769999999999999},"min":{"1min":0.27000000000000002,"5min":0.248,"15min":0.17299999999999999},"max":{"1min":11.111000000000001,"5min":13.581,"15min":20.108000000000001},"last":1.5}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2780086272,"data_stored":22190619850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4515266560,"data_stored":36075087223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32730,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1731653632,"data_stored":13747660871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:16:05.993 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:05.993 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:05.994 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:05.994 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:05.994 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:05.994 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:16:07.397 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:07.395+0000 7fc515f98700 -1 reset not still connected to 0x56221843d040 2023-10-03T18:16:08.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:08.768+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:08.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:08.835+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:09.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:09.030+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:10.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:10.903+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:10.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:10.915+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:10.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:10.915+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:11.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.177+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:16:11.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.233+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:11.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.233+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:11.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.327+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:11.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.327+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:11.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.533+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:11.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.629+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:11.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.629+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:11.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.673+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:11.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.673+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:11.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.681+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:11.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.856+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:16:11.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.887+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:11.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.899+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:11.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:12.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:11.995+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:12.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.000+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:12.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.003+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:12.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.003+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:12.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.003+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:12.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.009+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:12.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.009+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:12.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.065+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:16:12.074 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.073+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:12.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.074+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:12.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.099+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:12.132 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.130+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:12.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.132+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:12.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.224+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:12.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.224+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:12.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.224+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:12.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.347+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:12.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:12.347+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:13.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.116+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:13.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.117+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:13.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.406+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:13.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.805+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:13.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.805+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:13.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.805+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:13.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.806+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:13.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.844+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:13.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:13.844+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:17.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.132+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:17.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.132+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:17.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.141+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:17.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.161+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:17.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.210+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:17.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.307+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:17.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.307+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:17.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.379+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:17.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.379+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:17.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.419+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:17.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.622+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:17.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.652+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:17.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.652+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:17.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.674+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:17.676 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.674+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:17.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.704+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:17.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.704+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:17.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.897+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:17.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.897+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:17.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:17.932+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:20.674 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:16:20.672+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:16:22.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.703+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:22.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.703+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:22.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.703+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:22.831 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.828+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:22.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.855+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:22.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.882+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:22.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.882+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:22.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.882+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:22.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:22.923+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:23.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:23.015+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:23.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:23.991+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:24.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:23.993+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:24.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.375+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:24.405 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.403+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:24.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.482+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:24.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.549+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:24.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.566+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:24.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.566+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:24.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.579+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:24.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.580+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:24.810 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:24.803+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:25.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:16:26.957 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:16:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:16:26.991 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2023-10-03T18:16:24.970675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23018932017,"num_objects":48436,"num_object_clones":43170,"num_object_copies":96872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48436,"num_whiteouts":0,"num_read":1197966,"num_read_kb":4406257,"num_write":1109673,"num_write_kb":4429715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80416,"ondisk_log_size":80416,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13621168,"kb_used_data":8858840,"kb_used_omap":5697,"kb_used_meta":4756606,"kb_avail":269494352,"statfs":{"total":289910292480,"available":275962216448,"internally_reserved":0,"allocated":9071452160,"data_stored":75286173273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5834541,"internal_metadata":4870764755},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,2,10,12,6,25,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11251712,"num_objects":943,"num_object_clones":943,"num_object_copies":1886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":943,"num_whiteouts":0,"num_read":2644,"num_read_kb":10460,"num_write":2672,"num_write_kb":10688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001673"},"pg_stats":[{"pgid":"2.7","version":"229'158169","reported_seq":290083,"reported_epoch":229,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:20.761608+0000","last_change":"2023-10-03T18:16:19.174626+0000","last_active":"2023-10-03T18:16:20.761608+0000","last_peered":"2023-10-03T18:16:20.761608+0000","last_clean":"2023-10-03T18:16:20.761608+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:16:20.761608+0000","last_undegraded":"2023-10-03T18:16:20.761608+0000","last_fullsized":"2023-10-03T18:16:20.761608+0000","mapping_epoch":12,"log_start":"188'148133","ondisk_log_start":"188'148133","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":181,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:15:50.307258+0000","scrub_duration":505.38444223300002,"objects_trimmed":28,"snaptrim_duration":7.1318806859999997,"stat_sum":{"num_bytes":2868957670,"num_objects":8416,"num_object_clones":7765,"num_object_copies":16832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8416,"num_whiteouts":0,"num_read":146574,"num_read_kb":547969,"num_write":138113,"num_write_kb":551361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"}]},{"pgid":"2.6","version":"229'146302","reported_seq":268682,"reported_epoch":229,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:20.770902+0000","last_change":"2023-10-03T18:16:16.092470+0000","last_active":"2023-10-03T18:16:20.770902+0000","last_peered":"2023-10-03T18:16:20.770902+0000","last_clean":"2023-10-03T18:16:20.770902+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:16:20.770902+0000","last_undegraded":"2023-10-03T18:16:20.770902+0000","last_fullsized":"2023-10-03T18:16:20.770902+0000","mapping_epoch":12,"log_start":"187'136298","ondisk_log_start":"187'136298","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":100,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":38,"snaptrim_duration":4.0494858599999999,"stat_sum":{"num_bytes":2649404014,"num_objects":5479,"num_object_clones":4875,"num_object_copies":10958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5479,"num_whiteouts":0,"num_read":136760,"num_read_kb":507234,"num_write":127375,"num_write_kb":508697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"}]},{"pgid":"2.5","version":"229'175987","reported_seq":319867,"reported_epoch":229,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:20.742851+0000","last_change":"2023-10-03T18:16:19.145285+0000","last_active":"2023-10-03T18:16:20.742851+0000","last_peered":"2023-10-03T18:16:20.742851+0000","last_clean":"2023-10-03T18:16:20.742851+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:16:20.742851+0000","last_undegraded":"2023-10-03T18:16:20.742851+0000","last_fullsized":"2023-10-03T18:16:20.742851+0000","mapping_epoch":12,"log_start":"190'165891","ondisk_log_start":"190'165891","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":119,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":7.1029354199999997,"stat_sum":{"num_bytes":3106742978,"num_objects":7034,"num_object_clones":6323,"num_object_copies":14068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7034,"num_whiteouts":0,"num_read":165456,"num_read_kb":597408,"num_write":149935,"num_write_kb":597891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"1"}]},{"pgid":"2.0","version":"229'164569","reported_seq":302269,"reported_epoch":229,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:16:20.209484+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:16:20.209484+0000","last_peered":"2023-10-03T18:16:20.209484+0000","last_clean":"2023-10-03T18:16:20.209484+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:16:20.209484+0000","last_undegraded":"2023-10-03T18:16:20.209484+0000","last_fullsized":"2023-10-03T18:16:20.209484+0000","mapping_epoch":12,"log_start":"188'154545","ondisk_log_start":"188'154545","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":4441,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":180,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 515s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3001897716,"num_objects":8779,"num_object_clones":8092,"num_object_copies":17558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8779,"num_whiteouts":0,"num_read":152750,"num_read_kb":571803,"num_write":144881,"num_write_kb":578506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":388,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-03T18:16:06.995980+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:16:06.995980+0000","last_peered":"2023-10-03T18:16:06.995980+0000","last_clean":"2023-10-03T18:16:06.995980+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:16:06.995980+0000","last_undegraded":"2023-10-03T18:16:06.995980+0000","last_fullsized":"2023-10-03T18:16:06.995980+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"229'169363","reported_seq":308556,"reported_epoch":229,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:20.594880+0000","last_change":"2023-10-03T18:16:19.183282+0000","last_active":"2023-10-03T18:16:20.594880+0000","last_peered":"2023-10-03T18:16:20.594880+0000","last_clean":"2023-10-03T18:16:20.594880+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:16:20.594880+0000","last_undegraded":"2023-10-03T18:16:20.594880+0000","last_fullsized":"2023-10-03T18:16:20.594880+0000","mapping_epoch":12,"log_start":"189'159304","ondisk_log_start":"189'159304","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":113,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":1.960366547,"stat_sum":{"num_bytes":3053216585,"num_objects":6723,"num_object_clones":6024,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6723,"num_whiteouts":0,"num_read":155935,"num_read_kb":583054,"num_write":146906,"num_write_kb":586504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"7"}]},{"pgid":"2.1","version":"229'162195","reported_seq":288727,"reported_epoch":229,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:20.274449+0000","last_change":"2023-10-03T18:16:19.334186+0000","last_active":"2023-10-03T18:16:20.274449+0000","last_peered":"2023-10-03T18:16:20.274449+0000","last_clean":"2023-10-03T18:16:20.274449+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:16:20.274449+0000","last_undegraded":"2023-10-03T18:16:20.274449+0000","last_fullsized":"2023-10-03T18:16:20.274449+0000","mapping_epoch":12,"log_start":"192'152147","ondisk_log_start":"192'152147","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:35:21.640080+0000","scrub_duration":424.08306342600002,"objects_trimmed":133,"snaptrim_duration":7.3006218560000002,"stat_sum":{"num_bytes":2781835786,"num_objects":3050,"num_object_clones":2417,"num_object_copies":6100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3050,"num_whiteouts":0,"num_read":145141,"num_read_kb":530418,"num_write":133971,"num_write_kb":534488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"9"}]},{"pgid":"2.2","version":"229'162100","reported_seq":292651,"reported_epoch":229,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:21.069172+0000","last_change":"2023-10-03T18:16:18.147793+0000","last_active":"2023-10-03T18:16:21.069172+0000","last_peered":"2023-10-03T18:16:21.069172+0000","last_clean":"2023-10-03T18:16:21.069172+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:16:21.069172+0000","last_undegraded":"2023-10-03T18:16:21.069172+0000","last_fullsized":"2023-10-03T18:16:21.069172+0000","mapping_epoch":12,"log_start":"191'152062","ondisk_log_start":"191'152062","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:36:17.635837+0000","scrub_duration":370.574046178,"objects_trimmed":119,"snaptrim_duration":6.0977977250000004,"stat_sum":{"num_bytes":2784842500,"num_objects":3140,"num_object_clones":2498,"num_object_copies":6280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3140,"num_whiteouts":0,"num_read":152063,"num_read_kb":539429,"num_write":135067,"num_write_kb":538335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"7"}]},{"pgid":"2.4","version":"229'154609","reported_seq":282384,"reported_epoch":229,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:20.770252+0000","last_change":"2023-10-03T18:16:19.145473+0000","last_active":"2023-10-03T18:16:20.770252+0000","last_peered":"2023-10-03T18:16:20.770252+0000","last_clean":"2023-10-03T18:16:20.770252+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:16:20.770252+0000","last_undegraded":"2023-10-03T18:16:20.770252+0000","last_fullsized":"2023-10-03T18:16:20.770252+0000","mapping_epoch":12,"log_start":"188'144597","ondisk_log_start":"188'144597","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":104,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":1.314801688,"stat_sum":{"num_bytes":2771444400,"num_objects":5813,"num_object_clones":5176,"num_object_copies":11626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":143181,"num_read_kb":528851,"num_write":133288,"num_write_kb":532039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"b2","length":"1"},{"start":"b4","length":"1"},{"start":"b7","length":"1"},{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"7"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23018341649,"num_objects":48434,"num_object_clones":43170,"num_object_copies":96868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48434,"num_whiteouts":0,"num_read":1197860,"num_read_kb":4406166,"num_write":1109536,"num_write_kb":4427821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9064333312,"data_stored":75350354466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62035,"internal_metadata":0},"log_size":80317,"ondisk_log_size":80317,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738892,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2962036,"kb_used_data":1700844,"kb_used_omap":969,"kb_used_meta":1260214,"kb_avail":91409804,"statfs":{"total":96636764160,"available":93603639296,"internally_reserved":0,"allocated":1741664256,"data_stored":14390321220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1290460014},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":2.5800000000000001,"15min":2.4550000000000001},"min":{"1min":0.182,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":5.2350000000000003,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":1.893,"5min":2.3279999999999998,"15min":2.2639999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":10.340999999999999,"5min":10.994999999999999,"15min":16.285},"last":0.68200000000000005}]},{"osd":1,"last update":"Tue Oct 3 18:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":2.5419999999999998,"15min":2.4820000000000002},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":5.125,"5min":18.768999999999998,"15min":18.768999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.5369999999999999,"5min":2.3340000000000001,"15min":2.335},"min":{"1min":0.23799999999999999,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":4.8310000000000004,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.32200000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738891,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4098540,"kb_used_data":2729764,"kb_used_omap":2419,"kb_used_meta":1366348,"kb_avail":90273300,"statfs":{"total":96636764160,"available":92439859200,"internally_reserved":0,"allocated":2795278336,"data_stored":23264306919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1399140624},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,4,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2640000000000002,"5min":3.895,"15min":3.6309999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":31.474,"5min":31.474,"15min":31.474},"last":0.186},{"interface":"front","average":{"1min":4.6580000000000004,"5min":3.605,"15min":2.9700000000000002},"min":{"1min":0.24399999999999999,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":30.273,"5min":30.273,"15min":30.273},"last":0.249}]},{"osd":2,"last update":"Tue Oct 3 18:15:24 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":1.839,"15min":1.9690000000000001},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":5.4669999999999996,"5min":13.209,"15min":14.195},"last":1.627},{"interface":"front","average":{"1min":1.8520000000000001,"5min":2.0819999999999999,"15min":2.1120000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.214},"max":{"1min":7.0060000000000002,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.28299999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6560592,"kb_used_data":4428232,"kb_used_omap":2308,"kb_used_meta":2130043,"kb_avail":87811248,"statfs":{"total":96636764160,"available":89918717952,"internally_reserved":0,"allocated":4534509568,"data_stored":37631545134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2364331,"internal_metadata":2181164117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,6,8,5,22,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":2.79,"5min":2.4670000000000001,"15min":2.5649999999999999},"min":{"1min":0.20100000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":9.8680000000000003,"5min":10.529,"15min":16.997},"last":0.28599999999999998},{"interface":"front","average":{"1min":3.6179999999999999,"5min":2.8370000000000002,"15min":2.8029999999999999},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.298,"5min":13.516999999999999,"15min":20.033000000000001},"last":7.2050000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.774,"15min":2.3769999999999998},"min":{"1min":0.214,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":5.085,"5min":16.135999999999999,"15min":21.326000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":3.9830000000000001,"5min":3.1469999999999998,"15min":2.6179999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":10.817,"5min":13.581,"15min":20.108000000000001},"last":7.2569999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2792632320,"data_stored":23288728778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4532133888,"data_stored":37667276049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32730,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1739567104,"data_stored":14394349639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:16:26.993 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:26.993 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:26.993 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:26.993 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:26.994 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:26.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:16:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:16:27.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:16:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:16:28.569 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:16:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:16:29.280 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:16:29.957 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:16:29.970 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:16:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:16:30.686 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:16:31.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.625+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:31.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.625+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:31.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.634+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:31.637+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:33.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.826+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:33.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.909+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:33.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.909+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:33.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.909+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:33.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.917+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:33.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:33.928+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:34.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:34.037+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:35.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.939+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:35.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.953+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:35.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.953+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:35.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.953+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:35.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.953+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:35.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.966+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:35.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.966+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:35.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.966+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:35.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.966+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:35.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:35.966+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:36.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.587+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:36.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.621+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:36.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.621+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:36.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.687+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:36.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.712+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:36.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:36.736+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:38.276 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.276+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:38.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.390+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:38.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.534+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:38.537 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.534+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:38.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.556+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:38.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.576+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:38.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.576+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:38.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.576+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:38.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.636+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:38.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.660+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:38.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.660+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:38.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:38.670+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:41.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:41.916+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:41.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:41.950+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:41.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:41.952+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:41.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:41.952+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:41.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:41.952+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:42.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:42.022+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:42.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:42.047+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:43.388 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.385+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:43.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.397+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:43.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.397+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:43.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.397+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:43.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.397+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:43.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:43.397+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:49.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.134+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:49.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.134+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:49.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.134+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:49.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.134+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:49.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.153+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:49.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:49.240+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:50.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:16:51.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.166+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:51.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.166+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:51.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.166+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:51.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.166+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:51.236 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.232+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:51.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.354+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:51.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.369+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:51.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.369+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:51.409 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.407+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:51.542 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:16:51.558 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:16:51.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.565+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:51.576 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1545,"stamp":"2023-10-03T18:16:50.992907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23052232497,"num_objects":48822,"num_object_clones":43556,"num_object_copies":97644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48822,"num_whiteouts":0,"num_read":1207474,"num_read_kb":4439389,"num_write":1117737,"num_write_kb":4461971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80532,"ondisk_log_size":80532,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13840100,"kb_used_data":8922248,"kb_used_omap":6397,"kb_used_meta":4911426,"kb_avail":269275420,"statfs":{"total":289910292480,"available":275738030080,"internally_reserved":0,"allocated":9136381952,"data_stored":76172298665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6550997,"internal_metadata":5029300779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,8,9,10,10,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15630336,"num_objects":-45,"num_object_clones":-45,"num_object_copies":-90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-45,"num_whiteouts":0,"num_read":5276,"num_read_kb":16408,"num_write":3808,"num_write_kb":15232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005714"},"pg_stats":[{"pgid":"2.7","version":"231'159595","reported_seq":292291,"reported_epoch":231,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:47.336403+0000","last_change":"2023-10-03T18:16:44.634221+0000","last_active":"2023-10-03T18:16:47.336403+0000","last_peered":"2023-10-03T18:16:47.336403+0000","last_clean":"2023-10-03T18:16:47.336403+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:16:47.336403+0000","last_undegraded":"2023-10-03T18:16:47.336403+0000","last_fullsized":"2023-10-03T18:16:47.336403+0000","mapping_epoch":12,"log_start":"192'149543","ondisk_log_start":"192'149543","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":177,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:15:50.307258+0000","scrub_duration":505.38444223300002,"objects_trimmed":82,"snaptrim_duration":12.613936106000001,"stat_sum":{"num_bytes":2872779238,"num_objects":8501,"num_object_clones":7850,"num_object_copies":17002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8501,"num_whiteouts":0,"num_read":147719,"num_read_kb":552089,"num_write":139030,"num_write_kb":555029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"3"}]},{"pgid":"2.6","version":"231'147751","reported_seq":270848,"reported_epoch":231,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:47.315100+0000","last_change":"2023-10-03T18:16:46.790955+0000","last_active":"2023-10-03T18:16:47.315100+0000","last_peered":"2023-10-03T18:16:47.315100+0000","last_clean":"2023-10-03T18:16:47.315100+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:16:47.315100+0000","last_undegraded":"2023-10-03T18:16:47.315100+0000","last_fullsized":"2023-10-03T18:16:47.315100+0000","mapping_epoch":12,"log_start":"190'137702","ondisk_log_start":"190'137702","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":99,"snaptrim_duration":12.635802787999999,"stat_sum":{"num_bytes":2653090414,"num_objects":5518,"num_object_clones":4914,"num_object_copies":11036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":137834,"num_read_kb":511043,"num_write":128275,"num_write_kb":512297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"5"}]},{"pgid":"2.5","version":"231'177682","reported_seq":322347,"reported_epoch":231,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:47.315062+0000","last_change":"2023-10-03T18:16:46.790920+0000","last_active":"2023-10-03T18:16:47.315062+0000","last_peered":"2023-10-03T18:16:47.315062+0000","last_clean":"2023-10-03T18:16:47.315062+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:16:47.315062+0000","last_undegraded":"2023-10-03T18:16:47.315062+0000","last_fullsized":"2023-10-03T18:16:47.315062+0000","mapping_epoch":12,"log_start":"200'167599","ondisk_log_start":"200'167599","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":114,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":108,"snaptrim_duration":14.796340163,"stat_sum":{"num_bytes":3111162562,"num_objects":7106,"num_object_clones":6395,"num_object_copies":14212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7106,"num_whiteouts":0,"num_read":166746,"num_read_kb":601672,"num_write":150997,"num_write_kb":602139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"6"}]},{"pgid":"2.0","version":"231'166263","reported_seq":304937,"reported_epoch":231,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:16:50.366600+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:16:50.366600+0000","last_peered":"2023-10-03T18:16:50.366600+0000","last_clean":"2023-10-03T18:16:50.366600+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:16:50.366600+0000","last_undegraded":"2023-10-03T18:16:50.366600+0000","last_fullsized":"2023-10-03T18:16:50.366600+0000","mapping_epoch":12,"log_start":"193'156250","ondisk_log_start":"193'156250","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":4694,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":180,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 545s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3006821108,"num_objects":9026,"num_object_clones":8339,"num_object_copies":18052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9026,"num_whiteouts":0,"num_read":154036,"num_read_kb":576479,"num_write":146077,"num_write_kb":583290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":390,"reported_epoch":230,"state":"active+clean","last_fresh":"2023-10-03T18:16:22.285816+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:16:22.285816+0000","last_peered":"2023-10-03T18:16:22.285816+0000","last_clean":"2023-10-03T18:16:22.285816+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:16:22.285816+0000","last_undegraded":"2023-10-03T18:16:22.285816+0000","last_fullsized":"2023-10-03T18:16:22.285816+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"231'170959","reported_seq":310896,"reported_epoch":231,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:16:47.295348+0000","last_change":"2023-10-03T18:16:46.739704+0000","last_active":"2023-10-03T18:16:47.295348+0000","last_peered":"2023-10-03T18:16:47.295348+0000","last_clean":"2023-10-03T18:16:47.295348+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:16:47.295348+0000","last_undegraded":"2023-10-03T18:16:47.295348+0000","last_fullsized":"2023-10-03T18:16:47.295348+0000","mapping_epoch":12,"log_start":"198'160909","ondisk_log_start":"198'160909","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":109,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":80,"snaptrim_duration":11.250893301,"stat_sum":{"num_bytes":3057648457,"num_objects":6851,"num_object_clones":6152,"num_object_copies":13702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6851,"num_whiteouts":0,"num_read":157072,"num_read_kb":587166,"num_write":147986,"num_write_kb":590824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"2"}]},{"pgid":"2.1","version":"231'164227","reported_seq":291598,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-03T18:16:50.386542+0000","last_change":"2023-10-03T18:16:40.372360+0000","last_active":"2023-10-03T18:16:50.386542+0000","last_peered":"2023-10-03T18:16:50.386542+0000","last_clean":"2023-10-03T18:16:50.386542+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:16:50.386542+0000","last_undegraded":"2023-10-03T18:16:50.386542+0000","last_fullsized":"2023-10-03T18:16:50.386542+0000","mapping_epoch":12,"log_start":"201'154158","ondisk_log_start":"201'154158","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:35:21.640080+0000","scrub_duration":424.08306342600002,"objects_trimmed":149,"snaptrim_duration":8.3743977199999993,"stat_sum":{"num_bytes":2786578954,"num_objects":2967,"num_object_clones":2334,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":146472,"num_read_kb":535068,"num_write":135105,"num_write_kb":539024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"}]},{"pgid":"2.2","version":"231'163853","reported_seq":295127,"reported_epoch":231,"state":"active+clean","last_fresh":"2023-10-03T18:16:46.468279+0000","last_change":"2023-10-03T18:16:42.711644+0000","last_active":"2023-10-03T18:16:46.468279+0000","last_peered":"2023-10-03T18:16:46.468279+0000","last_clean":"2023-10-03T18:16:46.468279+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:16:46.468279+0000","last_undegraded":"2023-10-03T18:16:46.468279+0000","last_fullsized":"2023-10-03T18:16:46.468279+0000","mapping_epoch":12,"log_start":"201'153771","ondisk_log_start":"201'153771","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:36:17.635837+0000","scrub_duration":370.574046178,"objects_trimmed":196,"snaptrim_duration":10.705534354999999,"stat_sum":{"num_bytes":2788324100,"num_objects":2959,"num_object_clones":2317,"num_object_copies":5918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2959,"num_whiteouts":0,"num_read":153278,"num_read_kb":543332,"num_write":135916,"num_write_kb":541731,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"}]},{"pgid":"2.4","version":"231'156038","reported_seq":284487,"reported_epoch":231,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:16:47.284475+0000","last_change":"2023-10-03T18:16:46.722385+0000","last_active":"2023-10-03T18:16:47.284475+0000","last_peered":"2023-10-03T18:16:47.284475+0000","last_clean":"2023-10-03T18:16:47.284475+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:16:47.284475+0000","last_undegraded":"2023-10-03T18:16:47.284475+0000","last_fullsized":"2023-10-03T18:16:47.284475+0000","mapping_epoch":12,"log_start":"192'146003","ondisk_log_start":"192'146003","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":99,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":76,"snaptrim_duration":14.716289332000001,"stat_sum":{"num_bytes":2775237296,"num_objects":5892,"num_object_clones":5255,"num_object_copies":11784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":144211,"num_read_kb":532449,"num_write":134214,"num_write_kb":535743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23051642129,"num_objects":48820,"num_object_clones":43556,"num_object_copies":97640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48820,"num_whiteouts":0,"num_read":1207368,"num_read_kb":4439298,"num_write":1117600,"num_write_kb":4460077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9129332736,"data_stored":76182882850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62234,"internal_metadata":0},"log_size":80433,"ondisk_log_size":80433,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738898,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3061828,"kb_used_data":1715196,"kb_used_omap":969,"kb_used_meta":1345654,"kb_avail":91310012,"statfs":{"total":96636764160,"available":93501452288,"internally_reserved":0,"allocated":1756360704,"data_stored":14664865460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1377950574},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":2.5800000000000001,"15min":2.4550000000000001},"min":{"1min":0.182,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":5.2350000000000003,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.28000000000000003},{"interface":"front","average":{"1min":1.893,"5min":2.3279999999999998,"15min":2.2639999999999998},"min":{"1min":0.20899999999999999,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":10.340999999999999,"5min":10.994999999999999,"15min":16.285},"last":2.089}]},{"osd":1,"last update":"Tue Oct 3 18:16:08 2023","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":2.5419999999999998,"15min":2.4820000000000002},"min":{"1min":0.249,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":5.125,"5min":18.768999999999998,"15min":18.768999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":1.5369999999999999,"5min":2.3340000000000001,"15min":2.335},"min":{"1min":0.23799999999999999,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":4.8310000000000004,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738896,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4222712,"kb_used_data":2747052,"kb_used_omap":2419,"kb_used_meta":1473228,"kb_avail":90149128,"statfs":{"total":96636764160,"available":92312707072,"internally_reserved":0,"allocated":2812981248,"data_stored":23418510679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1508585744},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2,1,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6269999999999998,"5min":3.7949999999999999,"15min":3.597},"min":{"1min":0.186,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":13.814,"5min":31.474,"15min":31.474},"last":0.188},{"interface":"front","average":{"1min":2.6989999999999998,"5min":3.706,"15min":2.9489999999999998},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":8.2309999999999999,"5min":30.273,"15min":30.273},"last":0.27100000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6360000000000001,"5min":2.0950000000000002,"15min":2.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":10.388,"5min":13.209,"15min":14.195},"last":0.219},{"interface":"front","average":{"1min":1.4930000000000001,"5min":2.1469999999999998,"15min":2.0499999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.23300000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6555560,"kb_used_data":4460000,"kb_used_omap":3008,"kb_used_meta":2092543,"kb_avail":87816280,"statfs":{"total":96636764160,"available":89923870720,"internally_reserved":0,"allocated":4567040000,"data_stored":38088922526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3080787,"internal_metadata":2142764461},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,7,7,7,9,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":2.79,"5min":2.4670000000000001,"15min":2.5649999999999999},"min":{"1min":0.20100000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":9.8680000000000003,"5min":10.529,"15min":16.997},"last":0.20799999999999999},{"interface":"front","average":{"1min":3.6179999999999999,"5min":2.8370000000000002,"15min":2.8029999999999999},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.298,"5min":13.516999999999999,"15min":20.033000000000001},"last":1.5509999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.774,"15min":2.3769999999999998},"min":{"1min":0.214,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":5.085,"5min":16.135999999999999,"15min":21.326000000000001},"last":4.9740000000000002},{"interface":"front","average":{"1min":3.9830000000000001,"5min":3.1469999999999998,"15min":2.6179999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":10.817,"5min":13.581,"15min":20.108000000000001},"last":1.5900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2810916864,"data_stored":23423593674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4564201472,"data_stored":38088373521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32929,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1754214400,"data_stored":14670915655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:16:51.578 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:51.578 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:51.579 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:51.579 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:51.579 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:16:51.579 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:16:51.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.586+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:16:51.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.587+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:51.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.587+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:51.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.587+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:51.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.595+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:51.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.622+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:51.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.667+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:51.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.667+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:51.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.667+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:51.669 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.667+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:51.669 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.667+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:51.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.885+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:51.946 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.945+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:51.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.946+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:51.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.946+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:51.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.957+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:51.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.957+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:51.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.957+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:51.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:51.960+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:52.050 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:52.049+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:52.219 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:52.216+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:54.472 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.471+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:16:54.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.519+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:54.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.562+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:16:54.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.562+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:54.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.562+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:54.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.571+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:54.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.572+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:54.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.572+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:54.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.572+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:54.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.608+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:54.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:54.635+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:56.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.067+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:16:56.070 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.068+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:16:56.070 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.068+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:16:56.070 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.068+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:16:56.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.089+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:16:56.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.089+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:16:56.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.120+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:16:56.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.120+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:16:56.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:16:56.386+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:16:57.273 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:16:57.273+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:16:57.323 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:16:57.322+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:16:57.367 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:16:57.366+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:17:01.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.274+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:01.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.274+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:01.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:01.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:01.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:01.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:01.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:01.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:01.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.279+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:01.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.554+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:01.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:01.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:01.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:01.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:01.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:01.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:01.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:01.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:01.696+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:02.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.144+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:02.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.172+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:02.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.243+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:02.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.679+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:02.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.679+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:02.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.679+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:02.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.680+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:02.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.709+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:02.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.709+0000 7fc515f98700 -1 reset not still connected to 0x562207d2c4e0 2023-10-03T18:17:02.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.709+0000 7fc515f98700 -1 reset not still connected to 0x562217b49c70 2023-10-03T18:17:02.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.889+0000 7fc515f98700 -1 reset not still connected to 0x56221a1ec340 2023-10-03T18:17:02.966 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.966+0000 7fc515f98700 -1 reset not still connected to 0x56221b499450 2023-10-03T18:17:02.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.984+0000 7fc515f98700 -1 reset not still connected to 0x56221e45fee0 2023-10-03T18:17:02.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:02.984+0000 7fc515f98700 -1 reset not still connected to 0x562223df1d40 2023-10-03T18:17:03.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.011+0000 7fc515f98700 -1 reset not still connected to 0x56223290d930 2023-10-03T18:17:03.510 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.504+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:03.517 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.516+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:03.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.523+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:03.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.524+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:03.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.640+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:03.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.640+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:03.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.664+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:03.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.664+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:03.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.698+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:03.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.726+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:03.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.925+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:03.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:03.925+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:04.052 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.052+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:04.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.066+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:04.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.066+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:04.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.168+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:04.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.168+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:04.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.212+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:04.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.212+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:04.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.212+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:04.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.255+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:04.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.366+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:04.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.391+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:04.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.458+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:04.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.659+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:04.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.659+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:04.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.731+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:04.737 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.737+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:04.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.765+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:04.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.765+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:04.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.765+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:04.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.769+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:04.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:04.818+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:05.233 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.232+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:05.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.256+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:05.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.256+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:05.294 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.294+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:05.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.382+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:05.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:05.414+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:06.077 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.076+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:06.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.179+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:06.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.183+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:06.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.183+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:06.212 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.212+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:06.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.274+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:06.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:06.283+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:09.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.326+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:09.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.339+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:09.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.406+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:09.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.433+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:09.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.445+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:09.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.518+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:09.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.518+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:09.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.558+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:09.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.589+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:09.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.589+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:09.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.596+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:09.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.596+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:09.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.596+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:09.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.715+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:09.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.715+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:09.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.715+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:09.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:09.715+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:11.580 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:17:12.195 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:17:12.208 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:17:12.228 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1565,"stamp":"2023-10-03T18:17:11.003786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23079950129,"num_objects":49239,"num_object_clones":43973,"num_object_copies":98478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49239,"num_whiteouts":0,"num_read":1217259,"num_read_kb":4469128,"num_write":1124683,"num_write_kb":4489086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80515,"ondisk_log_size":80515,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13312660,"kb_used_data":8976352,"kb_used_omap":6399,"kb_used_meta":4329856,"kb_avail":269802860,"statfs":{"total":289910292480,"available":276278128640,"internally_reserved":0,"allocated":9191784448,"data_stored":76876520442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":4433772691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,5,2,9,8,21,9,18,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5398328,"num_objects":452,"num_object_clones":456,"num_object_copies":904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":452,"num_whiteouts":0,"num_read":2716,"num_read_kb":6246,"num_write":1438,"num_write_kb":5337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.527017"},"pg_stats":[{"pgid":"2.7","version":"240'160951","reported_seq":294434,"reported_epoch":240,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:09.263355+0000","last_change":"2023-10-03T18:17:07.412253+0000","last_active":"2023-10-03T18:17:09.263355+0000","last_peered":"2023-10-03T18:17:09.263355+0000","last_clean":"2023-10-03T18:17:09.263355+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:17:09.263355+0000","last_undegraded":"2023-10-03T18:17:09.263355+0000","last_fullsized":"2023-10-03T18:17:09.263355+0000","mapping_epoch":12,"log_start":"201'150949","ondisk_log_start":"201'150949","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":191,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:44:17.985965+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.86209072399999997,"stat_sum":{"num_bytes":2876346903,"num_objects":8533,"num_object_clones":7881,"num_object_copies":17066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8533,"num_whiteouts":0,"num_read":148914,"num_read_kb":555913,"num_write":139918,"num_write_kb":558506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"7"}]},{"pgid":"2.6","version":"240'148878","reported_seq":272740,"reported_epoch":240,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:09.218813+0000","last_change":"2023-10-03T18:17:07.266772+0000","last_active":"2023-10-03T18:17:09.218813+0000","last_peered":"2023-10-03T18:17:09.218813+0000","last_clean":"2023-10-03T18:17:09.218813+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:17:09.218813+0000","last_undegraded":"2023-10-03T18:17:09.218813+0000","last_fullsized":"2023-10-03T18:17:09.218813+0000","mapping_epoch":12,"log_start":"197'138805","ondisk_log_start":"197'138805","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":108,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.76121663500000003,"stat_sum":{"num_bytes":2656170606,"num_objects":5535,"num_object_clones":4931,"num_object_copies":11070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5535,"num_whiteouts":0,"num_read":138848,"num_read_kb":514372,"num_write":129034,"num_write_kb":515290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"}]},{"pgid":"2.5","version":"240'179187","reported_seq":324808,"reported_epoch":240,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:09.186166+0000","last_change":"2023-10-03T18:17:07.426665+0000","last_active":"2023-10-03T18:17:09.186166+0000","last_peered":"2023-10-03T18:17:09.186166+0000","last_clean":"2023-10-03T18:17:09.186166+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:17:09.186166+0000","last_undegraded":"2023-10-03T18:17:09.186166+0000","last_fullsized":"2023-10-03T18:17:09.186166+0000","mapping_epoch":12,"log_start":"203'169105","ondisk_log_start":"203'169105","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":129,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.35958194100000002,"stat_sum":{"num_bytes":3115242176,"num_objects":7175,"num_object_clones":6464,"num_object_copies":14350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7175,"num_whiteouts":0,"num_read":168317,"num_read_kb":606047,"num_write":152037,"num_write_kb":606159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"}]},{"pgid":"2.0","version":"241'167400","reported_seq":306818,"reported_epoch":241,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:17:10.438243+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:17:10.438243+0000","last_peered":"2023-10-03T18:17:10.438243+0000","last_clean":"2023-10-03T18:17:10.438243+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:17:10.438243+0000","last_undegraded":"2023-10-03T18:17:10.438243+0000","last_fullsized":"2023-10-03T18:17:10.438243+0000","mapping_epoch":12,"log_start":"201'157353","ondisk_log_start":"201'157353","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":4860,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":200,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 566s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3010044662,"num_objects":9198,"num_object_clones":8511,"num_object_copies":18396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9198,"num_whiteouts":0,"num_read":155142,"num_read_kb":579952,"num_write":146883,"num_write_kb":586435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":399,"reported_epoch":239,"state":"active+clean","last_fresh":"2023-10-03T18:17:07.671815+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:17:07.671815+0000","last_peered":"2023-10-03T18:17:07.671815+0000","last_clean":"2023-10-03T18:17:07.671815+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:17:07.671815+0000","last_undegraded":"2023-10-03T18:17:07.671815+0000","last_fullsized":"2023-10-03T18:17:07.671815+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"240'172336","reported_seq":313079,"reported_epoch":240,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:09.088224+0000","last_change":"2023-10-03T18:17:04.938834+0000","last_active":"2023-10-03T18:17:09.088224+0000","last_peered":"2023-10-03T18:17:09.088224+0000","last_clean":"2023-10-03T18:17:09.088224+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:17:09.088224+0000","last_undegraded":"2023-10-03T18:17:09.088224+0000","last_fullsized":"2023-10-03T18:17:09.088224+0000","mapping_epoch":12,"log_start":"202'162312","ondisk_log_start":"202'162312","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":123,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.45998667100000001,"stat_sum":{"num_bytes":3061392151,"num_objects":6902,"num_object_clones":6204,"num_object_copies":13804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6902,"num_whiteouts":0,"num_read":158284,"num_read_kb":591053,"num_write":148920,"num_write_kb":594484,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"6"}]},{"pgid":"2.1","version":"241'165453","reported_seq":293520,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:10.439022+0000","last_change":"2023-10-03T18:17:10.001366+0000","last_active":"2023-10-03T18:17:10.439022+0000","last_peered":"2023-10-03T18:17:10.439022+0000","last_clean":"2023-10-03T18:17:10.439022+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:17:10.439022+0000","last_undegraded":"2023-10-03T18:17:10.439022+0000","last_fullsized":"2023-10-03T18:17:10.439022+0000","mapping_epoch":12,"log_start":"204'155364","ondisk_log_start":"204'155364","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:38.575263+0000","scrub_duration":424.08306342600002,"objects_trimmed":16,"snaptrim_duration":0.47139507899999999,"stat_sum":{"num_bytes":2789552650,"num_objects":2992,"num_object_clones":2359,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":147687,"num_read_kb":538550,"num_write":135855,"num_write_kb":541930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"5"}]},{"pgid":"2.2","version":"238'165068","reported_seq":297191,"reported_epoch":238,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:06.531545+0000","last_change":"2023-10-03T18:17:06.155295+0000","last_active":"2023-10-03T18:17:06.531545+0000","last_peered":"2023-10-03T18:17:06.531545+0000","last_clean":"2023-10-03T18:17:06.531545+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:17:06.531545+0000","last_undegraded":"2023-10-03T18:17:06.531545+0000","last_fullsized":"2023-10-03T18:17:06.531545+0000","mapping_epoch":12,"log_start":"203'154977","ondisk_log_start":"203'154977","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:13:43.794373+0000","scrub_duration":370.574046178,"objects_trimmed":0,"snaptrim_duration":0.67088934700000002,"stat_sum":{"num_bytes":2791969490,"num_objects":2982,"num_object_clones":2341,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":154559,"num_read_kb":547009,"num_write":136831,"num_write_kb":545304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"3"}]},{"pgid":"2.4","version":"240'157315","reported_seq":286574,"reported_epoch":240,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:09.221277+0000","last_change":"2023-10-03T18:17:07.287432+0000","last_active":"2023-10-03T18:17:09.221277+0000","last_peered":"2023-10-03T18:17:09.221277+0000","last_clean":"2023-10-03T18:17:09.221277+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:17:09.221277+0000","last_undegraded":"2023-10-03T18:17:09.221277+0000","last_fullsized":"2023-10-03T18:17:09.221277+0000","mapping_epoch":12,"log_start":"201'147307","ondisk_log_start":"201'147307","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":114,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.88203795600000001,"stat_sum":{"num_bytes":2778641123,"num_objects":5920,"num_object_clones":5282,"num_object_copies":11840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5920,"num_whiteouts":0,"num_read":145402,"num_read_kb":536141,"num_write":135068,"num_write_kb":539084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23079359761,"num_objects":49237,"num_object_clones":43973,"num_object_copies":98474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49237,"num_whiteouts":0,"num_read":1217153,"num_read_kb":4469037,"num_write":1124546,"num_write_kb":4487192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9184415744,"data_stored":76999813921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80416,"ondisk_log_size":80416,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738902,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3101748,"kb_used_data":1724576,"kb_used_omap":969,"kb_used_meta":1376182,"kb_avail":91270092,"statfs":{"total":96636764160,"available":93460574208,"internally_reserved":0,"allocated":1765965824,"data_stored":14930052104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1409211246},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,0,1,2,0,10,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.899,"15min":2.5430000000000001},"min":{"1min":0.23400000000000001,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":13.859999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":1.214},{"interface":"front","average":{"1min":1.9890000000000001,"5min":2.3100000000000001,"15min":2.1760000000000002},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":11.471,"5min":11.471,"15min":16.285},"last":1.113}]},{"osd":1,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":2.4180000000000001,"15min":2.4180000000000001},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":6.4820000000000002,"5min":18.768999999999998,"15min":18.768999999999998},"last":1.155},{"interface":"front","average":{"1min":1.613,"5min":2.1080000000000001,"15min":2.2269999999999999},"min":{"1min":0.23000000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5019999999999998,"5min":18.751999999999999,"15min":19.097000000000001},"last":1.718}]}]},{"osd":0,"up_from":8,"seq":34359738900,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4303848,"kb_used_data":2764312,"kb_used_omap":2419,"kb_used_meta":1537100,"kb_avail":90067992,"statfs":{"total":96636764160,"available":92229623808,"internally_reserved":0,"allocated":2830655488,"data_stored":23490201852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1573990672},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,3,4,0,0,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6269999999999998,"5min":3.7949999999999999,"15min":3.597},"min":{"1min":0.186,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":13.814,"5min":31.474,"15min":31.474},"last":1.8089999999999999},{"interface":"front","average":{"1min":2.6989999999999998,"5min":3.706,"15min":2.9489999999999998},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":8.2309999999999999,"5min":30.273,"15min":30.273},"last":1.843}]},{"osd":2,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6360000000000001,"5min":2.0950000000000002,"15min":2.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":10.388,"5min":13.209,"15min":14.195},"last":0.35899999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":2.1469999999999998,"15min":2.0499999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.54500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738883,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5907064,"kb_used_data":4487464,"kb_used_omap":3010,"kb_used_meta":1416573,"kb_avail":88464776,"statfs":{"total":96636764160,"available":90587930624,"internally_reserved":0,"allocated":4595163136,"data_stored":38456266486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1450570773},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,0,4,4,11,9,13,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":2.79,"5min":2.4670000000000001,"15min":2.5649999999999999},"min":{"1min":0.20100000000000001,"5min":0.193,"15min":0.16300000000000001},"max":{"1min":9.8680000000000003,"5min":10.529,"15min":16.997},"last":0.34100000000000003},{"interface":"front","average":{"1min":3.6179999999999999,"5min":2.8370000000000002,"15min":2.8029999999999999},"min":{"1min":0.19500000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":12.298,"5min":13.516999999999999,"15min":20.033000000000001},"last":0.42299999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:16:14 2023","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.774,"15min":2.3769999999999998},"min":{"1min":0.214,"5min":0.20799999999999999,"15min":0.19400000000000001},"max":{"1min":5.085,"5min":16.135999999999999,"15min":21.326000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":3.9830000000000001,"5min":3.1469999999999998,"15min":2.6179999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":10.817,"5min":13.581,"15min":20.108000000000001},"last":0.31}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2828103680,"data_stored":23507709433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4592701440,"data_stored":38549402897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1763610624,"data_stored":14942701591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:17:12.229 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:12.230 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:12.230 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:12.230 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:12.230 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:12.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:17:14.902 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.898+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:14.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.912+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:14.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:14.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.928+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:14.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.928+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:14.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.987+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:14.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.987+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:14.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.987+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:14.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:14.987+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:15.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.010+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:15.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.017+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:15.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.017+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:15.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.017+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:15.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.017+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:15.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.047+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:15.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.090+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:15.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.090+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:15.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.090+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:15.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.090+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:15.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.090+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:15.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.385+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:15.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.386+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:15.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.386+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:15.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.386+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:15.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.412+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:15.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:15.463+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:21.018 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.013+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:21.019 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.013+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:21.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.155+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:21.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.167+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:21.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.167+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:21.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.199+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:21.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.199+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:21.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.199+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:21.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.279+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:21.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.374+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:21.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.722+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:21.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.758+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:21.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.805+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:21.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.840+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:21.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.840+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:21.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.840+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:21.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.840+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:21.844 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:21.840+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:22.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.249+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:22.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.425+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:22.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.425+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:22.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.435+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:22.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.435+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:22.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.435+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:22.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.435+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:22.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:22.435+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:24.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.118+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:24.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.127+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:24.144 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.140+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:24.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.196+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:24.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.224+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:24.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.224+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:24.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.224+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:24.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.245+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:24.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:24.291+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:27.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.108+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.148+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:27.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.164+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:27.298 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.296+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:27.298 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.297+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:27.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.297+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:27.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.297+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:27.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.297+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:27.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:27.305+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:28.909 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:28.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:17:28.909 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:28.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:17:28.909 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:28.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:17:28.910 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:28.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:17:28.910 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:28.901+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:17:28.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:28.931+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:29.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.010+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:29.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.012+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:29.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.012+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:29.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.038+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:29.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.041+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:29.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:29.041+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:30.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.107+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:30.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:30.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:30.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:30.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:30.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:30.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:30.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.173+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:30.272 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.271+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:30.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.271+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:30.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.344+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:30.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.378+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:30.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.391+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:30.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.391+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:30.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.426+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:30.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.494+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:30.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:30.533+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:31.625 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.622+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:31.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.695+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:31.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.700+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:31.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.700+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:31.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.700+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:31.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.875+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:31.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.876+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:31.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.876+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:31.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.876+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:31.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.918+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:31.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.924+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:31.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.960+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:31.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:31.960+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:32.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.002+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:32.034 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.033+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:32.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.089+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:32.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:17:32.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.459+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:32.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.471+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:32.505 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.503+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:32.613 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.612+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:32.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.619+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:32.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.624+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:32.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.650+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:32.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:32.655+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:32.959 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:17:32.987 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:17:33.007 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1575,"stamp":"2023-10-03T18:17:31.027255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23101429553,"num_objects":50871,"num_object_clones":45605,"num_object_copies":101742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50871,"num_whiteouts":0,"num_read":1222246,"num_read_kb":4488499,"num_write":1129895,"num_write_kb":4509883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80471,"ondisk_log_size":80471,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13428944,"kb_used_data":9016952,"kb_used_omap":6399,"kb_used_meta":4405568,"kb_avail":269686576,"statfs":{"total":289910292480,"available":276159053824,"internally_reserved":0,"allocated":9233358848,"data_stored":80073245237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":4511301779},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,1,11,12,18,17,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15867904,"num_objects":963,"num_object_clones":963,"num_object_copies":1926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":963,"num_whiteouts":0,"num_read":3580,"num_read_kb":14276,"num_write":3831,"num_write_kb":15324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.009387"},"pg_stats":[{"pgid":"2.7","version":"241'162128","reported_seq":295870,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:30.956863+0000","last_change":"2023-10-03T18:17:28.759809+0000","last_active":"2023-10-03T18:17:30.956863+0000","last_peered":"2023-10-03T18:17:30.956863+0000","last_clean":"2023-10-03T18:17:30.956863+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:17:30.956863+0000","last_undegraded":"2023-10-03T18:17:30.956863+0000","last_fullsized":"2023-10-03T18:17:30.956863+0000","mapping_epoch":12,"log_start":"204'152051","ondisk_log_start":"204'152051","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":190,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:43:09.435956+0000","scrub_duration":505.38444223300002,"objects_trimmed":89,"snaptrim_duration":19.239153225999999,"stat_sum":{"num_bytes":2878984727,"num_objects":8779,"num_object_clones":8127,"num_object_copies":17558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8779,"num_whiteouts":0,"num_read":149526,"num_read_kb":558357,"num_write":140558,"num_write_kb":561066,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"}]},{"pgid":"2.6","version":"241'150026","reported_seq":274094,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:30.950115+0000","last_change":"2023-10-03T18:17:30.197445+0000","last_active":"2023-10-03T18:17:30.950115+0000","last_peered":"2023-10-03T18:17:30.950115+0000","last_clean":"2023-10-03T18:17:30.950115+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:17:30.950115+0000","last_undegraded":"2023-10-03T18:17:30.950115+0000","last_fullsized":"2023-10-03T18:17:30.950115+0000","mapping_epoch":12,"log_start":"202'140011","ondisk_log_start":"202'140011","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":107,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":92,"snaptrim_duration":20.676086676000001,"stat_sum":{"num_bytes":2658714222,"num_objects":5773,"num_object_clones":5169,"num_object_copies":11546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5773,"num_whiteouts":0,"num_read":139411,"num_read_kb":516624,"num_write":129655,"num_write_kb":517774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"}]},{"pgid":"2.5","version":"241'180498","reported_seq":326469,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:30.888034+0000","last_change":"2023-10-03T18:17:30.472884+0000","last_active":"2023-10-03T18:17:30.888034+0000","last_peered":"2023-10-03T18:17:30.888034+0000","last_clean":"2023-10-03T18:17:30.888034+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:17:30.888034+0000","last_undegraded":"2023-10-03T18:17:30.888034+0000","last_fullsized":"2023-10-03T18:17:30.888034+0000","mapping_epoch":12,"log_start":"206'170407","ondisk_log_start":"206'170407","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":128,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":87,"snaptrim_duration":18.324432554000001,"stat_sum":{"num_bytes":3118187200,"num_objects":7450,"num_object_clones":6739,"num_object_copies":14900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7450,"num_whiteouts":0,"num_read":169052,"num_read_kb":608953,"num_write":152755,"num_write_kb":609020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"3"}]},{"pgid":"2.0","version":"241'168530","reported_seq":308231,"reported_epoch":241,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:17:30.704154+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:17:30.704154+0000","last_peered":"2023-10-03T18:17:30.704154+0000","last_clean":"2023-10-03T18:17:30.704154+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:17:30.704154+0000","last_undegraded":"2023-10-03T18:17:30.704154+0000","last_fullsized":"2023-10-03T18:17:30.704154+0000","mapping_epoch":12,"log_start":"203'158457","ondisk_log_start":"203'158457","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5008,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":200,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 586s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3012711158,"num_objects":9528,"num_object_clones":8841,"num_object_copies":19056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9528,"num_whiteouts":0,"num_read":155730,"num_read_kb":582296,"num_write":147534,"num_write_kb":589039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2023-10-03T18:17:08.502160+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:17:08.502160+0000","last_peered":"2023-10-03T18:17:08.502160+0000","last_clean":"2023-10-03T18:17:08.502160+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:17:08.502160+0000","last_undegraded":"2023-10-03T18:17:08.502160+0000","last_fullsized":"2023-10-03T18:17:08.502160+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"241'173656","reported_seq":314649,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:30.811907+0000","last_change":"2023-10-03T18:17:30.213730+0000","last_active":"2023-10-03T18:17:30.811907+0000","last_peered":"2023-10-03T18:17:30.811907+0000","last_clean":"2023-10-03T18:17:30.811907+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:17:30.811907+0000","last_undegraded":"2023-10-03T18:17:30.811907+0000","last_fullsized":"2023-10-03T18:17:30.811907+0000","mapping_epoch":12,"log_start":"205'163617","ondisk_log_start":"205'163617","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":122,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":112,"snaptrim_duration":16.350652370999999,"stat_sum":{"num_bytes":3064345367,"num_objects":7147,"num_object_clones":6449,"num_object_copies":14294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7147,"num_whiteouts":0,"num_read":158926,"num_read_kb":593617,"num_write":149631,"num_write_kb":597328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"9"}]},{"pgid":"2.1","version":"241'167017","reported_seq":295234,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:30.699194+0000","last_change":"2023-10-03T18:17:29.321190+0000","last_active":"2023-10-03T18:17:30.699194+0000","last_peered":"2023-10-03T18:17:30.699194+0000","last_clean":"2023-10-03T18:17:30.699194+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:17:30.699194+0000","last_undegraded":"2023-10-03T18:17:30.699194+0000","last_fullsized":"2023-10-03T18:17:30.699194+0000","mapping_epoch":12,"log_start":"212'156976","ondisk_log_start":"212'156976","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:38.575263+0000","scrub_duration":424.08306342600002,"objects_trimmed":331,"snaptrim_duration":19.791245869000001,"stat_sum":{"num_bytes":2792100362,"num_objects":2982,"num_object_clones":2349,"num_object_copies":5964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2982,"num_whiteouts":0,"num_read":148250,"num_read_kb":540794,"num_write":136456,"num_write_kb":544334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"5"}]},{"pgid":"2.2","version":"241'166599","reported_seq":298875,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:26.657289+0000","last_change":"2023-10-03T18:17:26.356909+0000","last_active":"2023-10-03T18:17:26.657289+0000","last_peered":"2023-10-03T18:17:26.657289+0000","last_clean":"2023-10-03T18:17:26.657289+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:17:26.657289+0000","last_undegraded":"2023-10-03T18:17:26.657289+0000","last_fullsized":"2023-10-03T18:17:26.657289+0000","mapping_epoch":12,"log_start":"207'156587","ondisk_log_start":"207'156587","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:59:24.536316+0000","scrub_duration":370.574046178,"objects_trimmed":245,"snaptrim_duration":16.857705359000001,"stat_sum":{"num_bytes":2794418898,"num_objects":3021,"num_object_clones":2380,"num_object_copies":6042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3021,"num_whiteouts":0,"num_read":155258,"num_read_kb":549298,"num_write":137433,"num_write_kb":547672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"2"}]},{"pgid":"2.4","version":"241'158539","reported_seq":288010,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:30.924723+0000","last_change":"2023-10-03T18:17:30.471630+0000","last_active":"2023-10-03T18:17:30.924723+0000","last_peered":"2023-10-03T18:17:30.924723+0000","last_clean":"2023-10-03T18:17:30.924723+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:17:30.924723+0000","last_undegraded":"2023-10-03T18:17:30.924723+0000","last_fullsized":"2023-10-03T18:17:30.924723+0000","mapping_epoch":12,"log_start":"203'148515","ondisk_log_start":"203'148515","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":112,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":81,"snaptrim_duration":20.939386487,"stat_sum":{"num_bytes":2781377251,"num_objects":6189,"num_object_clones":5551,"num_object_copies":12378,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6189,"num_whiteouts":0,"num_read":145987,"num_read_kb":538469,"num_write":135736,"num_write_kb":541756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"8"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23100839185,"num_objects":50869,"num_object_clones":45605,"num_object_copies":101738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50869,"num_whiteouts":0,"num_read":1222140,"num_read_kb":4488408,"num_write":1129758,"num_write_kb":4507989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9226420224,"data_stored":80111991330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80372,"ondisk_log_size":80372,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738906,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3173036,"kb_used_data":1732068,"kb_used_omap":969,"kb_used_meta":1439990,"kb_avail":91198804,"statfs":{"total":96636764160,"available":93387575296,"internally_reserved":0,"allocated":1773637632,"data_stored":15444669448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1474550638},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.899,"15min":2.5430000000000001},"min":{"1min":0.23400000000000001,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":13.859999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.79900000000000004},{"interface":"front","average":{"1min":1.9890000000000001,"5min":2.3100000000000001,"15min":2.1760000000000002},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":11.471,"5min":11.471,"15min":16.285},"last":0.26600000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":2.4180000000000001,"15min":2.4180000000000001},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":6.4820000000000002,"5min":18.768999999999998,"15min":18.768999999999998},"last":0.307},{"interface":"front","average":{"1min":1.613,"5min":2.1080000000000001,"15min":2.2269999999999999},"min":{"1min":0.23000000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5019999999999998,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.76600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738904,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4401760,"kb_used_data":2776152,"kb_used_omap":2419,"kb_used_meta":1623180,"kb_avail":89970080,"statfs":{"total":96636764160,"available":92129361920,"internally_reserved":0,"allocated":2842779648,"data_stored":24550132491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1662136592},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,1,0,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6269999999999998,"5min":3.7949999999999999,"15min":3.597},"min":{"1min":0.186,"5min":0.17699999999999999,"15min":0.17299999999999999},"max":{"1min":13.814,"5min":31.474,"15min":31.474},"last":4.6470000000000002},{"interface":"front","average":{"1min":2.6989999999999998,"5min":3.706,"15min":2.9489999999999998},"min":{"1min":0.249,"5min":0.22500000000000001,"15min":0.186},"max":{"1min":8.2309999999999999,"5min":30.273,"15min":30.273},"last":1.04}]},{"osd":2,"last update":"Tue Oct 3 18:16:25 2023","interfaces":[{"interface":"back","average":{"1min":2.6360000000000001,"5min":2.0950000000000002,"15min":2.0129999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.182},"max":{"1min":10.388,"5min":13.209,"15min":14.195},"last":0.28499999999999998},{"interface":"front","average":{"1min":1.4930000000000001,"5min":2.1469999999999998,"15min":2.0499999999999998},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.0190000000000001,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.23699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738887,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5854148,"kb_used_data":4508732,"kb_used_omap":3010,"kb_used_meta":1342397,"kb_avail":88517692,"statfs":{"total":96636764160,"available":90642116608,"internally_reserved":0,"allocated":4616941568,"data_stored":40078443298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1374614549},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,0,9,9,11,15,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.2250000000000001,"15min":2.4849999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":12.531000000000001,"5min":12.531000000000001,"15min":16.997},"last":7.4219999999999997},{"interface":"front","average":{"1min":2.4049999999999998,"5min":2.9550000000000001,"15min":2.7050000000000001},"min":{"1min":0.27000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.6020000000000003,"5min":13.516999999999999,"15min":20.033000000000001},"last":7.9169999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.6859999999999999,"15min":2.1859999999999999},"min":{"1min":0.22,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":6.4569999999999999,"5min":16.135999999999999,"15min":21.326000000000001},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.5609999999999999,"5min":3.2589999999999999,"15min":2.6240000000000001},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":12.337999999999999,"5min":13.581,"15min":20.108000000000001},"last":0.246}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2840215552,"data_stored":24568450298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4614725632,"data_stored":40090731793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1771479040,"data_stored":15452809239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:17:33.009 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:33.009 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:33.009 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:33.009 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:33.010 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:33.010 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:17:34.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.519+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:34.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.519+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:34.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.521+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:34.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.521+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:34.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.568+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:34.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.597+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:34.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.597+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:34.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.608+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:34.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.639+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:34.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.639+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:34.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.639+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:34.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.639+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:34.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.639+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:34.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.692+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:34.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.692+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:34.772 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.771+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:34.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.830+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:34.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.830+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:34.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.830+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:34.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.830+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:34.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.830+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:34.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.839+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:17:34.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.848+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:34.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:34.848+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:35.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.010+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:35.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.010+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:35.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.014+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:35.015 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.014+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:35.015 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.014+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:35.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.046+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:35.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.046+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:35.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:35.046+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:37.339 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:37.338+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:17:37.339 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:37.338+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:17:37.339 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:37.338+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:17:37.339 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:37.338+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:17:37.339 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:37.338+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:17:38.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.068+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:38.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.069+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:38.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.069+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:38.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.101+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:38.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.101+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:38.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.101+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:38.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:38.175+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:39.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:39.804+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:39.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:39.853+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:39.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:39.882+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:39.961 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:39.959+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:40.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.046+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:40.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.046+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:40.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.116+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:40.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.116+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:40.147 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.147+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:40.486 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.485+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:40.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.522+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:40.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.797+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:40.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.797+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:40.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.797+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:40.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.820+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:40.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.821+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:40.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:40.821+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:42.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.257+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:42.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.257+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:42.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.257+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:42.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.257+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:42.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.257+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:42.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.309+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:42.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.309+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:42.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.662+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:42.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.723+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:42.733 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:42.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.733+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:42.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.954+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:42.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:42.954+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:43.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:43.046+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:43.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:43.046+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:43.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:43.046+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:43.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:43.087+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:43.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:43.087+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:44.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.370+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:44.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.370+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:44.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.394+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:44.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.396+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:44.397 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.396+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:44.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.483+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:44.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.483+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:44.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.490+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:44.506 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:17:44.503+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:17:44.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.531+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:44.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:44.547+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:46.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.450+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:46.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.542+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:46.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.542+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:46.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.620+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:46.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.681+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:46.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:46.690+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:51.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:51.350+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:51.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:51.350+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:51.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:51.362+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:51.387 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:51.384+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:53.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:17:53.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:53.453+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:53.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:53.454+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:53.468 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:53.466+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:53.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:53.500+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:53.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:53.569+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:53.765 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:17:53.766 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:17:53.787 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1586,"stamp":"2023-10-03T18:17:53.041646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23130580785,"num_objects":51069,"num_object_clones":45803,"num_object_copies":102138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51069,"num_whiteouts":0,"num_read":1230211,"num_read_kb":4515562,"num_write":1136928,"num_write_kb":4538015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80459,"ondisk_log_size":80459,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13730548,"kb_used_data":9075740,"kb_used_omap":6399,"kb_used_meta":4648384,"kb_avail":269384972,"statfs":{"total":289910292480,"available":275850211328,"internally_reserved":0,"allocated":9293557760,"data_stored":80789840437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":4759945363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,6,21,18,12,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20967424,"num_objects":150,"num_object_clones":150,"num_object_copies":300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":150,"num_whiteouts":0,"num_read":6144,"num_read_kb":19827,"num_write":5060,"num_write_kb":20240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004820"},"pg_stats":[{"pgid":"2.7","version":"241'163384","reported_seq":297781,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:52.463322+0000","last_change":"2023-10-03T18:17:50.358891+0000","last_active":"2023-10-03T18:17:52.463322+0000","last_peered":"2023-10-03T18:17:52.463322+0000","last_clean":"2023-10-03T18:17:52.463322+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:17:52.463322+0000","last_undegraded":"2023-10-03T18:17:52.463322+0000","last_fullsized":"2023-10-03T18:17:52.463322+0000","mapping_epoch":12,"log_start":"207'153354","ondisk_log_start":"207'153354","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":187,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:43:09.435956+0000","scrub_duration":505.38444223300002,"objects_trimmed":210,"snaptrim_duration":39.075156212000003,"stat_sum":{"num_bytes":2882474519,"num_objects":8795,"num_object_clones":8143,"num_object_copies":17590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8795,"num_whiteouts":0,"num_read":150468,"num_read_kb":561669,"num_write":141405,"num_write_kb":564454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"3"}]},{"pgid":"2.6","version":"241'151228","reported_seq":275883,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:52.414142+0000","last_change":"2023-10-03T18:17:50.106246+0000","last_active":"2023-10-03T18:17:52.414142+0000","last_peered":"2023-10-03T18:17:52.414142+0000","last_clean":"2023-10-03T18:17:52.414142+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:17:52.414142+0000","last_undegraded":"2023-10-03T18:17:52.414142+0000","last_fullsized":"2023-10-03T18:17:52.414142+0000","mapping_epoch":12,"log_start":"205'141214","ondisk_log_start":"205'141214","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":104,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":179,"snaptrim_duration":40.584894128999998,"stat_sum":{"num_bytes":2662351470,"num_objects":5824,"num_object_clones":5220,"num_object_copies":11648,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":140261,"num_read_kb":519541,"num_write":130530,"num_write_kb":521274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"}]},{"pgid":"2.5","version":"241'181894","reported_seq":328661,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:52.430043+0000","last_change":"2023-10-03T18:17:50.313515+0000","last_active":"2023-10-03T18:17:52.430043+0000","last_peered":"2023-10-03T18:17:52.430043+0000","last_clean":"2023-10-03T18:17:52.430043+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:17:52.430043+0000","last_undegraded":"2023-10-03T18:17:52.430043+0000","last_fullsized":"2023-10-03T18:17:52.430043+0000","mapping_epoch":12,"log_start":"215'171811","ondisk_log_start":"215'171811","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":124,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":215,"snaptrim_duration":40.791516891999997,"stat_sum":{"num_bytes":3122115264,"num_objects":7457,"num_object_clones":6746,"num_object_copies":14914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7457,"num_whiteouts":0,"num_read":170231,"num_read_kb":612809,"num_write":153700,"num_write_kb":612800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"7"}]},{"pgid":"2.0","version":"241'169701","reported_seq":310095,"reported_epoch":241,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:17:50.838458+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:17:50.838458+0000","last_peered":"2023-10-03T18:17:50.838458+0000","last_clean":"2023-10-03T18:17:50.838458+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:17:50.838458+0000","last_undegraded":"2023-10-03T18:17:50.838458+0000","last_fullsized":"2023-10-03T18:17:50.838458+0000","mapping_epoch":12,"log_start":"206'159661","ondisk_log_start":"206'159661","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5182,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":200,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 606s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3016180470,"num_objects":9670,"num_object_clones":8983,"num_object_copies":19340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9670,"num_whiteouts":0,"num_read":156659,"num_read_kb":585548,"num_write":148381,"num_write_kb":592427,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":401,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-03T18:17:09.521713+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:17:09.521713+0000","last_peered":"2023-10-03T18:17:09.521713+0000","last_clean":"2023-10-03T18:17:09.521713+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:17:09.521713+0000","last_undegraded":"2023-10-03T18:17:09.521713+0000","last_fullsized":"2023-10-03T18:17:09.521713+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"241'175054","reported_seq":316738,"reported_epoch":241,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:17:52.456097+0000","last_change":"2023-10-03T18:17:51.546217+0000","last_active":"2023-10-03T18:17:52.456097+0000","last_peered":"2023-10-03T18:17:52.456097+0000","last_clean":"2023-10-03T18:17:52.456097+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:17:52.456097+0000","last_undegraded":"2023-10-03T18:17:52.456097+0000","last_fullsized":"2023-10-03T18:17:52.456097+0000","mapping_epoch":12,"log_start":"214'165022","ondisk_log_start":"214'165022","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":118,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":243,"snaptrim_duration":42.025050055999998,"stat_sum":{"num_bytes":3068318487,"num_objects":7165,"num_object_clones":6467,"num_object_copies":14330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7165,"num_whiteouts":0,"num_read":159915,"num_read_kb":597137,"num_write":150588,"num_write_kb":601156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"2"}]},{"pgid":"2.1","version":"241'168222","reported_seq":297037,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-03T18:17:50.809934+0000","last_change":"2023-10-03T18:17:38.908723+0000","last_active":"2023-10-03T18:17:50.809934+0000","last_peered":"2023-10-03T18:17:50.809934+0000","last_clean":"2023-10-03T18:17:50.809934+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:17:50.809934+0000","last_undegraded":"2023-10-03T18:17:50.809934+0000","last_fullsized":"2023-10-03T18:17:50.809934+0000","mapping_epoch":12,"log_start":"215'158178","ondisk_log_start":"215'158178","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:49:38.575263+0000","scrub_duration":424.08306342600002,"objects_trimmed":464,"snaptrim_duration":29.378941262000001,"stat_sum":{"num_bytes":2795385354,"num_objects":2980,"num_object_clones":2347,"num_object_copies":5960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2980,"num_whiteouts":0,"num_read":149173,"num_read_kb":543844,"num_write":137237,"num_write_kb":547458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"}]},{"pgid":"2.2","version":"241'168177","reported_seq":301211,"reported_epoch":241,"state":"active+clean","last_fresh":"2023-10-03T18:17:51.718687+0000","last_change":"2023-10-03T18:17:41.508413+0000","last_active":"2023-10-03T18:17:51.718687+0000","last_peered":"2023-10-03T18:17:51.718687+0000","last_clean":"2023-10-03T18:17:51.718687+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:17:51.718687+0000","last_undegraded":"2023-10-03T18:17:51.718687+0000","last_fullsized":"2023-10-03T18:17:51.718687+0000","mapping_epoch":12,"log_start":"215'158097","ondisk_log_start":"215'158097","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:59:24.536316+0000","scrub_duration":370.574046178,"objects_trimmed":472,"snaptrim_duration":32.009590760999998,"stat_sum":{"num_bytes":2798322386,"num_objects":2970,"num_object_clones":2329,"num_object_copies":5940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2970,"num_whiteouts":0,"num_read":156462,"num_read_kb":553185,"num_write":138373,"num_write_kb":551432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"}]},{"pgid":"2.4","version":"241'159757","reported_seq":289901,"reported_epoch":241,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:17:52.422593+0000","last_change":"2023-10-03T18:17:51.549769+0000","last_active":"2023-10-03T18:17:52.422593+0000","last_peered":"2023-10-03T18:17:52.422593+0000","last_clean":"2023-10-03T18:17:52.422593+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:17:52.422593+0000","last_undegraded":"2023-10-03T18:17:52.422593+0000","last_fullsized":"2023-10-03T18:17:52.422593+0000","mapping_epoch":12,"log_start":"206'149720","ondisk_log_start":"206'149720","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":109,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":191,"snaptrim_duration":39.264892047000004,"stat_sum":{"num_bytes":2784842467,"num_objects":6206,"num_object_clones":5568,"num_object_copies":12412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6206,"num_whiteouts":0,"num_read":146936,"num_read_kb":541738,"num_write":136577,"num_write_kb":545120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23129990417,"num_objects":51067,"num_object_clones":45803,"num_object_copies":102134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51067,"num_whiteouts":0,"num_read":1230105,"num_read_kb":4515471,"num_write":1136791,"num_write_kb":4536121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9285591040,"data_stored":80781593122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80360,"ondisk_log_size":80360,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738910,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3210836,"kb_used_data":1742092,"kb_used_omap":969,"kb_used_meta":1467766,"kb_avail":91161004,"statfs":{"total":96636764160,"available":93348868096,"internally_reserved":0,"allocated":1783902208,"data_stored":15625319432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1502993262},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":3.532,"5min":2.899,"15min":2.5430000000000001},"min":{"1min":0.23400000000000001,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":13.859999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.9890000000000001,"5min":2.3100000000000001,"15min":2.1760000000000002},"min":{"1min":0.25800000000000001,"5min":0.20100000000000001,"15min":0.17199999999999999},"max":{"1min":11.471,"5min":11.471,"15min":16.285},"last":0.79200000000000004}]},{"osd":1,"last update":"Tue Oct 3 18:17:08 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":2.4180000000000001,"15min":2.4180000000000001},"min":{"1min":0.248,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":6.4820000000000002,"5min":18.768999999999998,"15min":18.768999999999998},"last":0.82699999999999996},{"interface":"front","average":{"1min":1.613,"5min":2.1080000000000001,"15min":2.2269999999999999},"min":{"1min":0.23000000000000001,"5min":0.19700000000000001,"15min":0.188},"max":{"1min":6.5019999999999998,"5min":18.751999999999999,"15min":19.097000000000001},"last":0.49299999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738909,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4550500,"kb_used_data":2795932,"kb_used_omap":2419,"kb_used_meta":1752140,"kb_avail":89821340,"statfs":{"total":96636764160,"available":91977052160,"internally_reserved":0,"allocated":2863034368,"data_stored":24772233995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1794191632},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:17:27 2023","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":3.327,"15min":3.46},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":14.458,"5min":31.474,"15min":31.474},"last":0.32100000000000001},{"interface":"front","average":{"1min":2.3700000000000001,"5min":3.3180000000000001,"15min":2.9129999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.186},"max":{"1min":9.6359999999999992,"5min":30.273,"15min":30.273},"last":7.0519999999999996}]},{"osd":2,"last update":"Tue Oct 3 18:17:27 2023","interfaces":[{"interface":"back","average":{"1min":2.069,"5min":2.0960000000000001,"15min":2.008},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":10.974,"5min":13.209,"15min":14.195},"last":0.28100000000000003},{"interface":"front","average":{"1min":2.0550000000000002,"5min":2.1469999999999998,"15min":2.0249999999999999},"min":{"1min":0.23300000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.1560000000000006,"5min":16.033999999999999,"15min":16.033999999999999},"last":7.0949999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738891,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5969212,"kb_used_data":4537716,"kb_used_omap":3010,"kb_used_meta":1428477,"kb_avail":88402628,"statfs":{"total":96636764160,"available":90524291072,"internally_reserved":0,"allocated":4646621184,"data_stored":40392287010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1462760469},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3,3,14,15,8,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.2250000000000001,"15min":2.4849999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":12.531000000000001,"5min":12.531000000000001,"15min":16.997},"last":0.185},{"interface":"front","average":{"1min":2.4049999999999998,"5min":2.9550000000000001,"15min":2.7050000000000001},"min":{"1min":0.27000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.6020000000000003,"5min":13.516999999999999,"15min":20.033000000000001},"last":0.60299999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.6859999999999999,"15min":2.1859999999999999},"min":{"1min":0.22,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":6.4569999999999999,"5min":16.135999999999999,"15min":21.326000000000001},"last":5.8890000000000002},{"interface":"front","average":{"1min":2.5609999999999999,"5min":3.2589999999999999,"15min":2.6240000000000001},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":12.337999999999999,"5min":13.581,"15min":20.108000000000001},"last":0.30299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2860167168,"data_stored":24767294714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4643651584,"data_stored":40391210257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1781772288,"data_stored":15623088151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:17:53.789 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:53.789 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:53.789 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:53.789 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:53.790 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:17:53.790 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:17:54.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:54.897+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:54.898 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:54.897+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:54.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:54.939+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:54.967 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:54.967+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:54.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:54.969+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:55.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.183+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:55.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.183+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:55.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.188+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:55.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.309+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:55.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.382+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:55.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.402+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:55.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.403+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:55.417 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.417+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:55.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.434+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:55.441 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:55.437+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:58.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:58.749+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:58.776 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:58.775+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:58.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:58.930+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:58.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:58.957+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:17:59.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.538+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:17:59.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.560+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:17:59.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.578+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:17:59.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.626+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:17:59.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.627+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:17:59.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.641+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:17:59.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.641+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:17:59.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.713+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:17:59.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.717+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:17:59.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:17:59.717+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.410+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.410+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:00.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.430+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:00.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.475+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:00.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.475+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:00.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.475+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:00.563 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.563+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:00.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.591+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:00.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.591+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:00.754 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.753+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:00.776 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.774+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:00.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.777+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:00.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.795+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:00.893 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.893+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:00.893 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.893+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:00.909 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.906+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:00.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.906+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:00.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:00.910+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:02.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:02.723+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:02.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:02.772+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:02.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:02.865+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:02.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:02.915+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:02.979 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:02.978+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:03.088 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.087+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:03.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.107+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:03.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.187+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:03.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.299+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:03.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.299+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:03.300 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.299+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:03.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.305+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:03.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.305+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:03.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.329+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:03.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.330+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:03.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.330+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:03.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.332+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:03.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.459+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:03.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.459+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:03.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.539+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:03.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.539+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:03.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.661+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:03.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.690+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:03.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.774+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:03.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.774+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:03.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.838+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:03.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.857+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:03.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:03.869+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:07.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.091+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:07.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.323+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:07.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.328+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:07.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.328+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:07.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.341+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:07.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.373+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:07.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.373+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:07.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.419+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:07.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.425+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:07.461 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:07.461+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:09.331 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:09.329+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:18:09.332 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:09.330+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:18:09.332 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:09.330+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:18:09.332 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:09.330+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:18:09.332 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:09.330+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:18:13.792 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:18:14.412 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:18:14.419 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:18:14.437 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1603,"stamp":"2023-10-03T18:18:13.056654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23152908081,"num_objects":51788,"num_object_clones":46522,"num_object_copies":103576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51788,"num_whiteouts":0,"num_read":1238617,"num_read_kb":4539330,"num_write":1142531,"num_write_kb":4559707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80396,"ondisk_log_size":80396,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14059816,"kb_used_data":9119756,"kb_used_omap":6399,"kb_used_meta":4933632,"kb_avail":269055704,"statfs":{"total":289910292480,"available":275513040896,"internally_reserved":0,"allocated":9338630144,"data_stored":82254231431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":5052039315},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,5,0,10,16,15,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6787072,"num_objects":772,"num_object_clones":772,"num_object_copies":1544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":772,"num_whiteouts":0,"num_read":3188,"num_read_kb":7867,"num_write":1775,"num_write_kb":6613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.766673"},"pg_stats":[{"pgid":"2.7","version":"248'164383","reported_seq":299282,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:08.807995+0000","last_change":"2023-10-03T18:18:06.090128+0000","last_active":"2023-10-03T18:18:08.807995+0000","last_peered":"2023-10-03T18:18:08.807995+0000","last_clean":"2023-10-03T18:18:08.807995+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:18:08.807995+0000","last_undegraded":"2023-10-03T18:18:08.807995+0000","last_fullsized":"2023-10-03T18:18:08.807995+0000","mapping_epoch":12,"log_start":"215'154357","ondisk_log_start":"215'154357","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":204,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:29:05.694901+0000","scrub_duration":505.38444223300002,"objects_trimmed":12,"snaptrim_duration":0.085500434,"stat_sum":{"num_bytes":2885059044,"num_objects":8871,"num_object_clones":8220,"num_object_copies":17742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8871,"num_whiteouts":0,"num_read":151365,"num_read_kb":564322,"num_write":142048,"num_write_kb":566950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"6"}]},{"pgid":"2.6","version":"248'152124","reported_seq":277338,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:08.800339+0000","last_change":"2023-10-03T18:18:02.665245+0000","last_active":"2023-10-03T18:18:08.800339+0000","last_peered":"2023-10-03T18:18:08.800339+0000","last_clean":"2023-10-03T18:18:08.800339+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:18:08.800339+0000","last_undegraded":"2023-10-03T18:18:08.800339+0000","last_fullsized":"2023-10-03T18:18:08.800339+0000","mapping_epoch":12,"log_start":"211'142116","ondisk_log_start":"211'142116","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":121,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.34895149199999997,"stat_sum":{"num_bytes":2664760020,"num_objects":5891,"num_object_clones":5285,"num_object_copies":11782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5891,"num_whiteouts":0,"num_read":141073,"num_read_kb":522058,"num_write":131133,"num_write_kb":523637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"1"}]},{"pgid":"2.5","version":"248'183036","reported_seq":330435,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:08.772444+0000","last_change":"2023-10-03T18:18:06.059699+0000","last_active":"2023-10-03T18:18:08.772444+0000","last_peered":"2023-10-03T18:18:08.772444+0000","last_clean":"2023-10-03T18:18:08.772444+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:18:08.772444+0000","last_undegraded":"2023-10-03T18:18:08.772444+0000","last_fullsized":"2023-10-03T18:18:08.772444+0000","mapping_epoch":12,"log_start":"217'173014","ondisk_log_start":"217'173014","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":141,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.75093166700000002,"stat_sum":{"num_bytes":3124916877,"num_objects":7533,"num_object_clones":6823,"num_object_copies":15066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7533,"num_whiteouts":0,"num_read":171487,"num_read_kb":615954,"num_write":154429,"num_write_kb":615568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"}]},{"pgid":"2.0","version":"248'170903","reported_seq":311987,"reported_epoch":248,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:18:10.986821+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:18:10.986821+0000","last_peered":"2023-10-03T18:18:10.986821+0000","last_clean":"2023-10-03T18:18:10.986821+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:18:10.986821+0000","last_undegraded":"2023-10-03T18:18:10.986821+0000","last_fullsized":"2023-10-03T18:18:10.986821+0000","mapping_epoch":12,"log_start":"215'160864","ondisk_log_start":"215'160864","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5365,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":220,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 626s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3019436687,"num_objects":9886,"num_object_clones":9201,"num_object_copies":19772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9886,"num_whiteouts":0,"num_read":157751,"num_read_kb":588965,"num_write":149187,"num_write_kb":595578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":408,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-03T18:18:06.329588+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:18:06.329588+0000","last_peered":"2023-10-03T18:18:06.329588+0000","last_clean":"2023-10-03T18:18:06.329588+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:18:06.329588+0000","last_undegraded":"2023-10-03T18:18:06.329588+0000","last_fullsized":"2023-10-03T18:18:06.329588+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"248'176121","reported_seq":318308,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:08.717165+0000","last_change":"2023-10-03T18:18:05.497553+0000","last_active":"2023-10-03T18:18:08.717165+0000","last_peered":"2023-10-03T18:18:08.717165+0000","last_clean":"2023-10-03T18:18:08.717165+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:18:08.717165+0000","last_undegraded":"2023-10-03T18:18:08.717165+0000","last_fullsized":"2023-10-03T18:18:08.717165+0000","mapping_epoch":12,"log_start":"216'166025","ondisk_log_start":"216'166025","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":135,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.181305786,"stat_sum":{"num_bytes":3071103716,"num_objects":7250,"num_object_clones":6553,"num_object_copies":14500,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7250,"num_whiteouts":0,"num_read":160823,"num_read_kb":599841,"num_write":151284,"num_write_kb":603871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"5"}]},{"pgid":"2.1","version":"248'169445","reported_seq":298907,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:11.005195+0000","last_change":"2023-10-03T18:18:10.724401+0000","last_active":"2023-10-03T18:18:11.005195+0000","last_peered":"2023-10-03T18:18:11.005195+0000","last_clean":"2023-10-03T18:18:11.005195+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:18:11.005195+0000","last_undegraded":"2023-10-03T18:18:11.005195+0000","last_fullsized":"2023-10-03T18:18:11.005195+0000","mapping_epoch":12,"log_start":"217'159380","ondisk_log_start":"217'159380","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:59.704648+0000","scrub_duration":424.08306342600002,"objects_trimmed":59,"snaptrim_duration":3.3863860130000001,"stat_sum":{"num_bytes":2798363248,"num_objects":3022,"num_object_clones":2387,"num_object_copies":6044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3022,"num_whiteouts":0,"num_read":150363,"num_read_kb":547214,"num_write":137976,"num_write_kb":550314,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"5"}]},{"pgid":"2.2","version":"248'169425","reported_seq":303100,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:11.753485+0000","last_change":"2023-10-03T18:18:10.860733+0000","last_active":"2023-10-03T18:18:11.753485+0000","last_peered":"2023-10-03T18:18:11.753485+0000","last_clean":"2023-10-03T18:18:11.753485+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:18:11.753485+0000","last_undegraded":"2023-10-03T18:18:11.753485+0000","last_fullsized":"2023-10-03T18:18:11.753485+0000","mapping_epoch":12,"log_start":"218'159402","ondisk_log_start":"218'159402","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:54:13.755597+0000","scrub_duration":370.574046178,"objects_trimmed":62,"snaptrim_duration":3.53403519,"stat_sum":{"num_bytes":2801312517,"num_objects":3053,"num_object_clones":2411,"num_object_copies":6106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":157787,"num_read_kb":556514,"num_write":139135,"num_write_kb":554354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"4"}]},{"pgid":"2.4","version":"248'160741","reported_seq":291421,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:08.668828+0000","last_change":"2023-10-03T18:18:05.522697+0000","last_active":"2023-10-03T18:18:08.668828+0000","last_peered":"2023-10-03T18:18:08.668828+0000","last_clean":"2023-10-03T18:18:08.668828+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:18:08.668828+0000","last_undegraded":"2023-10-03T18:18:08.668828+0000","last_fullsized":"2023-10-03T18:18:08.668828+0000","mapping_epoch":12,"log_start":"215'150723","ondisk_log_start":"215'150723","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":126,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":51.372912710999998,"stat_sum":{"num_bytes":2787365604,"num_objects":6280,"num_object_clones":5642,"num_object_copies":12560,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6280,"num_whiteouts":0,"num_read":147862,"num_read_kb":544371,"num_write":137202,"num_write_kb":547541,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23152317713,"num_objects":51786,"num_object_clones":46522,"num_object_copies":103572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51786,"num_whiteouts":0,"num_read":1238511,"num_read_kb":4539239,"num_write":1142394,"num_write_kb":4557813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9330642944,"data_stored":82375338530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80297,"ondisk_log_size":80297,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738914,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3282752,"kb_used_data":1751544,"kb_used_omap":969,"kb_used_meta":1530230,"kb_avail":91089088,"statfs":{"total":96636764160,"available":93275226112,"internally_reserved":0,"allocated":1793581056,"data_stored":15957738906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1566956398},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,2,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":2.6960000000000002,"15min":2.5840000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":8.2469999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":1.855},{"interface":"front","average":{"1min":3.7810000000000001,"5min":2.5979999999999999,"15min":2.2149999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.17199999999999999},"max":{"1min":30.128,"5min":30.128,"15min":30.128},"last":3.214}]},{"osd":1,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.4300000000000002,"15min":2.4470000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":17.274000000000001,"5min":17.274000000000001,"15min":18.768999999999998},"last":3.2639999999999998},{"interface":"front","average":{"1min":1.508,"5min":1.8180000000000001,"15min":2.1680000000000001},"min":{"1min":0.29699999999999999,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":6.1719999999999997,"5min":8.7349999999999994,"15min":19.097000000000001},"last":0.503}]}]},{"osd":0,"up_from":8,"seq":34359738913,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4668252,"kb_used_data":2810964,"kb_used_omap":2419,"kb_used_meta":1854860,"kb_avail":89703588,"statfs":{"total":96636764160,"available":91856474112,"internally_reserved":0,"allocated":2878427136,"data_stored":25343909125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1899376912},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,0,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:17:27 2023","interfaces":[{"interface":"back","average":{"1min":2.9390000000000001,"5min":3.327,"15min":3.46},"min":{"1min":0.14799999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":14.458,"5min":31.474,"15min":31.474},"last":10.66},{"interface":"front","average":{"1min":2.3700000000000001,"5min":3.3180000000000001,"15min":2.9129999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.186},"max":{"1min":9.6359999999999992,"5min":30.273,"15min":30.273},"last":10.24}]},{"osd":2,"last update":"Tue Oct 3 18:17:27 2023","interfaces":[{"interface":"back","average":{"1min":2.069,"5min":2.0960000000000001,"15min":2.008},"min":{"1min":0.192,"5min":0.192,"15min":0.182},"max":{"1min":10.974,"5min":13.209,"15min":14.195},"last":0.45200000000000001},{"interface":"front","average":{"1min":2.0550000000000002,"5min":2.1469999999999998,"15min":2.0249999999999999},"min":{"1min":0.23300000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":9.1560000000000006,"5min":16.033999999999999,"15min":16.033999999999999},"last":3.5979999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738894,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108812,"kb_used_data":4557248,"kb_used_omap":3010,"kb_used_meta":1548541,"kb_avail":88263028,"statfs":{"total":96636764160,"available":90381340672,"internally_reserved":0,"allocated":4666621952,"data_stored":40952583400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1585706005},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,4,0,7,12,10,4,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.9139999999999999,"5min":2.2250000000000001,"15min":2.4849999999999999},"min":{"1min":0.183,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":12.531000000000001,"5min":12.531000000000001,"15min":16.997},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.4049999999999998,"5min":2.9550000000000001,"15min":2.7050000000000001},"min":{"1min":0.27000000000000002,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":9.6020000000000003,"5min":13.516999999999999,"15min":20.033000000000001},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:17:16 2023","interfaces":[{"interface":"back","average":{"1min":1.724,"5min":1.6859999999999999,"15min":2.1859999999999999},"min":{"1min":0.22,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":6.4569999999999999,"5min":16.135999999999999,"15min":21.326000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":2.5609999999999999,"5min":3.2589999999999999,"15min":2.6240000000000001},"min":{"1min":0.25700000000000001,"5min":0.23999999999999999,"15min":0.20699999999999999},"max":{"1min":12.337999999999999,"5min":13.581,"15min":20.108000000000001},"last":10.093999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2875326464,"data_stored":25349487918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4664045568,"data_stored":41034368273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1791270912,"data_stored":15991482339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:18:14.439 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:14.439 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:14.439 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:14.439 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:14.440 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:14.440 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:18:14.684 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.675+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:14.751 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.747+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:14.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.751+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:14.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.782+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:14.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.807+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:14.809 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.807+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:14.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:14.892+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:15.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.458+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:15.551 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.549+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:15.551 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.549+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:15.563 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.562+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:15.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.586+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:15.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.586+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:15.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.586+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:15.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.586+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:15.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.747+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:15.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:15.785+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:17.157 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:17.153+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:18:17.238 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:17.237+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:18:17.247 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:17.246+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:18:18.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:18.836+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:18.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:18.843+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:18.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:18.843+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:18.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:18.843+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:18.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:18.899+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:20.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:20.985+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:20.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:20.985+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:20.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:20.985+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:20.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:20.985+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:20.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:20.992+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:21.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:21.040+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:22.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.143+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:22.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.143+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:22.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.159+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:22.200 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.199+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:22.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.220+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:22.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.220+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:22.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.220+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:22.222 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.220+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:22.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:22.248+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:23.563 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.562+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:23.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.634+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:23.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.635+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:23.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.635+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:23.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.670+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:23.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.723+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:23.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.741+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:23.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.741+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:23.794 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.792+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:23.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.869+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:23.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:23.900+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:24.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.002+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:24.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.073+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:24.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.073+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:24.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.073+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:24.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.093+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:24.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:24.152+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:25.102 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.101+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:25.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.101+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:25.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.104+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:25.105 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.104+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:25.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.104+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:25.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.195+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:25.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:25.195+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:28.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:28.984+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:28.986 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:28.984+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:29.019 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:29.016+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:29.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:29.019+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:29.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:29.060+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:29.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:29.060+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:31.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.002+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:31.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.054+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:31.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.054+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:31.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.182+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:31.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.182+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:31.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.300+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:31.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.300+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:31.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.300+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:31.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.418+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:31.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.424+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:31.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.424+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:31.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.424+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:31.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.507+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:31.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:31.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.511+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:31.538 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.536+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:31.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.627+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:31.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.635+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:31.677 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.674+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:31.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.674+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:31.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.675+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:31.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.675+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:31.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.696+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:31.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:31.930+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:32.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.000+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:32.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.063+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:32.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.107+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:32.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.113+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:32.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.133+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.136+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.138+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:32.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.287+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:32.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.287+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:32.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.287+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:32.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.287+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:32.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.298+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:32.303 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:32.301+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:33.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.195+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:33.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.258+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:33.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.276+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:33.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.276+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:33.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.276+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:33.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.284+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:33.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:33.284+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:34.243 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:18:34.238+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:18:34.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:18:35.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.183+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:35.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.207+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:35.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.241+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:35.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.241+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:35.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.260+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:35.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.333+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:35.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.333+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:35.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.333+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:35.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.339+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:35.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.379+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:35.459 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:18:35.463 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:18:35.483 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1614,"stamp":"2023-10-03T18:18:35.069763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23175771953,"num_objects":53310,"num_object_clones":48044,"num_object_copies":106620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53310,"num_whiteouts":0,"num_read":1243914,"num_read_kb":4560334,"num_write":1147978,"num_write_kb":4581495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80583,"ondisk_log_size":80583,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14391524,"kb_used_data":9162888,"kb_used_omap":6399,"kb_used_meta":5222208,"kb_avail":268723996,"statfs":{"total":289910292480,"available":275173371904,"internally_reserved":0,"allocated":9382797312,"data_stored":85170005895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":5347541139},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,4,10,9,20,17,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11976704,"num_objects":470,"num_object_clones":470,"num_object_copies":940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":0,"num_read":2719,"num_read_kb":10780,"num_write":2849,"num_write_kb":11396,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.011273"},"pg_stats":[{"pgid":"2.7","version":"248'165592","reported_seq":300850,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:30.999971+0000","last_change":"2023-10-03T18:18:28.605529+0000","last_active":"2023-10-03T18:18:30.999971+0000","last_peered":"2023-10-03T18:18:30.999971+0000","last_clean":"2023-10-03T18:18:30.999971+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:18:30.999971+0000","last_undegraded":"2023-10-03T18:18:30.999971+0000","last_fullsized":"2023-10-03T18:18:30.999971+0000","mapping_epoch":12,"log_start":"218'155562","ondisk_log_start":"218'155562","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":200,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:29:05.694901+0000","scrub_duration":505.38444223300002,"objects_trimmed":97,"snaptrim_duration":21.235268474000002,"stat_sum":{"num_bytes":2887918052,"num_objects":9122,"num_object_clones":8471,"num_object_copies":18244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9122,"num_whiteouts":0,"num_read":152074,"num_read_kb":567146,"num_write":142731,"num_write_kb":569682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"}]},{"pgid":"2.6","version":"248'153289","reported_seq":278803,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:30.938842+0000","last_change":"2023-10-03T18:18:27.019762+0000","last_active":"2023-10-03T18:18:30.938842+0000","last_peered":"2023-10-03T18:18:30.938842+0000","last_clean":"2023-10-03T18:18:30.938842+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:18:30.938842+0000","last_undegraded":"2023-10-03T18:18:30.938842+0000","last_fullsized":"2023-10-03T18:18:30.938842+0000","mapping_epoch":12,"log_start":"216'143218","ondisk_log_start":"216'143218","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":118,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":98,"snaptrim_duration":19.656905912999999,"stat_sum":{"num_bytes":2667389652,"num_objects":6118,"num_object_clones":5512,"num_object_copies":12236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6118,"num_whiteouts":0,"num_read":141728,"num_read_kb":524646,"num_write":131767,"num_write_kb":526173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"2"}]},{"pgid":"2.5","version":"248'184368","reported_seq":332101,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:30.855294+0000","last_change":"2023-10-03T18:18:28.615888+0000","last_active":"2023-10-03T18:18:30.855294+0000","last_peered":"2023-10-03T18:18:30.855294+0000","last_clean":"2023-10-03T18:18:30.855294+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:18:30.855294+0000","last_undegraded":"2023-10-03T18:18:30.855294+0000","last_fullsized":"2023-10-03T18:18:30.855294+0000","mapping_epoch":12,"log_start":"220'174319","ondisk_log_start":"220'174319","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":139,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":82,"snaptrim_duration":16.798255404999999,"stat_sum":{"num_bytes":3128148621,"num_objects":7819,"num_object_clones":7109,"num_object_copies":15638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7819,"num_whiteouts":0,"num_read":172210,"num_read_kb":618806,"num_write":155193,"num_write_kb":618624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"1"}]},{"pgid":"2.0","version":"248'172057","reported_seq":313464,"reported_epoch":248,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:18:31.113362+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:18:31.113362+0000","last_peered":"2023-10-03T18:18:31.113362+0000","last_clean":"2023-10-03T18:18:31.113362+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:18:31.113362+0000","last_undegraded":"2023-10-03T18:18:31.113362+0000","last_fullsized":"2023-10-03T18:18:31.113362+0000","mapping_epoch":12,"log_start":"218'161968","ondisk_log_start":"218'161968","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5511,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":220,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 646s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3022418575,"num_objects":10196,"num_object_clones":9511,"num_object_copies":20392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10196,"num_whiteouts":0,"num_read":158366,"num_read_kb":591397,"num_write":149886,"num_write_kb":598374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":408,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-03T18:18:06.329588+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:18:06.329588+0000","last_peered":"2023-10-03T18:18:06.329588+0000","last_clean":"2023-10-03T18:18:06.329588+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:18:06.329588+0000","last_undegraded":"2023-10-03T18:18:06.329588+0000","last_fullsized":"2023-10-03T18:18:06.329588+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"248'177481","reported_seq":319954,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:30.945027+0000","last_change":"2023-10-03T18:18:29.103460+0000","last_active":"2023-10-03T18:18:30.945027+0000","last_peered":"2023-10-03T18:18:30.945027+0000","last_clean":"2023-10-03T18:18:30.945027+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:18:30.945027+0000","last_undegraded":"2023-10-03T18:18:30.945027+0000","last_fullsized":"2023-10-03T18:18:30.945027+0000","mapping_epoch":12,"log_start":"220'167429","ondisk_log_start":"220'167429","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":132,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":115,"snaptrim_duration":18.473934152999998,"stat_sum":{"num_bytes":3074253540,"num_objects":7493,"num_object_clones":6796,"num_object_copies":14986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7493,"num_whiteouts":0,"num_read":161500,"num_read_kb":602541,"num_write":152038,"num_write_kb":606887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"8"}]},{"pgid":"2.1","version":"248'170980","reported_seq":300625,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:31.153067+0000","last_change":"2023-10-03T18:18:29.923211+0000","last_active":"2023-10-03T18:18:31.153067+0000","last_peered":"2023-10-03T18:18:31.153067+0000","last_clean":"2023-10-03T18:18:31.153067+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:18:31.153067+0000","last_undegraded":"2023-10-03T18:18:31.153067+0000","last_fullsized":"2023-10-03T18:18:31.153067+0000","mapping_epoch":12,"log_start":"224'160890","ondisk_log_start":"224'160890","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:59.704648+0000","scrub_duration":424.08306342600002,"objects_trimmed":353,"snaptrim_duration":22.580576876999999,"stat_sum":{"num_bytes":2800996976,"num_objects":3034,"num_object_clones":2399,"num_object_copies":6068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3034,"num_whiteouts":0,"num_read":150953,"num_read_kb":549558,"num_write":138606,"num_write_kb":552834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"1"}]},{"pgid":"2.2","version":"248'170928","reported_seq":304864,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:32.072286+0000","last_change":"2023-10-03T18:18:30.397253+0000","last_active":"2023-10-03T18:18:32.072286+0000","last_peered":"2023-10-03T18:18:32.072286+0000","last_clean":"2023-10-03T18:18:32.072286+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:18:32.072286+0000","last_undegraded":"2023-10-03T18:18:32.072286+0000","last_fullsized":"2023-10-03T18:18:32.072286+0000","mapping_epoch":12,"log_start":"226'160911","ondisk_log_start":"226'160911","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:54:13.755597+0000","scrub_duration":370.574046178,"objects_trimmed":349,"snaptrim_duration":23.064167446999999,"stat_sum":{"num_bytes":2804007685,"num_objects":3045,"num_object_clones":2403,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":158408,"num_read_kb":558982,"num_write":139779,"num_write_kb":556930,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"2"}]},{"pgid":"2.4","version":"248'161911","reported_seq":292969,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:30.810806+0000","last_change":"2023-10-03T18:18:29.096949+0000","last_active":"2023-10-03T18:18:30.810806+0000","last_peered":"2023-10-03T18:18:30.810806+0000","last_clean":"2023-10-03T18:18:30.810806+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:18:30.810806+0000","last_undegraded":"2023-10-03T18:18:30.810806+0000","last_fullsized":"2023-10-03T18:18:30.810806+0000","mapping_epoch":12,"log_start":"217'151825","ondisk_log_start":"217'151825","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":122,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":122,"snaptrim_duration":21.725270428000002,"stat_sum":{"num_bytes":2790048484,"num_objects":6481,"num_object_clones":5843,"num_object_copies":12962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6481,"num_whiteouts":0,"num_read":148569,"num_read_kb":547167,"num_write":137841,"num_write_kb":550097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23175181585,"num_objects":53308,"num_object_clones":48044,"num_object_copies":106616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53308,"num_whiteouts":0,"num_read":1243808,"num_read_kb":4560243,"num_write":1147841,"num_write_kb":4579601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9374744576,"data_stored":85159156258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80484,"ondisk_log_size":80484,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738918,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3342108,"kb_used_data":1759508,"kb_used_omap":969,"kb_used_meta":1581622,"kb_avail":91029732,"statfs":{"total":96636764160,"available":93214445568,"internally_reserved":0,"allocated":1801736192,"data_stored":16495228314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1619581806},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,2,4,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":2.6960000000000002,"15min":2.5840000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":8.2469999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":2.4449999999999998},{"interface":"front","average":{"1min":3.7810000000000001,"5min":2.5979999999999999,"15min":2.2149999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.17199999999999999},"max":{"1min":30.128,"5min":30.128,"15min":30.128},"last":2.3620000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.4300000000000002,"15min":2.4470000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":17.274000000000001,"5min":17.274000000000001,"15min":18.768999999999998},"last":2.3999999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.8180000000000001,"15min":2.1680000000000001},"min":{"1min":0.29699999999999999,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":6.1719999999999997,"5min":8.7349999999999994,"15min":19.097000000000001},"last":0.28399999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738917,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4751432,"kb_used_data":2823488,"kb_used_omap":2419,"kb_used_meta":1925516,"kb_avail":89620408,"statfs":{"total":96636764160,"available":91771297792,"internally_reserved":0,"allocated":2891251712,"data_stored":26095054085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":1971728656},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,1,2,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.0800000000000001,"15min":3.3849999999999998},"min":{"1min":0.19800000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":10.66,"5min":31.474,"15min":31.474},"last":0.26400000000000001},{"interface":"front","average":{"1min":4.3419999999999996,"5min":3.5299999999999998,"15min":3.032},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":19.498999999999999,"5min":30.273,"15min":30.273},"last":0.33100000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.996,"15min":1.8460000000000001},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":8.5449999999999999,"5min":13.209,"15min":13.209},"last":0.28199999999999997},{"interface":"front","average":{"1min":2.028,"5min":2.1459999999999999,"15min":2.0350000000000001},"min":{"1min":0.251,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.383,"5min":16.033999999999999,"15min":16.033999999999999},"last":0.35799999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738898,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6297984,"kb_used_data":4579892,"kb_used_omap":3010,"kb_used_meta":1715069,"kb_avail":88073856,"statfs":{"total":96636764160,"available":90187628544,"internally_reserved":0,"allocated":4689809408,"data_stored":42579723496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1756230677},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4,3,6,14,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":3.8900000000000001,"5min":2.573,"15min":2.556},"min":{"1min":0.185,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":18.613,"5min":18.613,"15min":18.613},"last":3.004},{"interface":"front","average":{"1min":3.363,"5min":2.9119999999999999,"15min":2.722},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":21.498999999999999,"5min":21.498999999999999,"15min":21.498999999999999},"last":0.39500000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":1.6399999999999999,"15min":2.1749999999999998},"min":{"1min":0.221,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":8.3580000000000005,"5min":13.948,"15min":21.326000000000001},"last":0.28399999999999997},{"interface":"front","average":{"1min":2.7770000000000001,"5min":3.0310000000000001,"15min":2.6640000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":10.093999999999999,"5min":12.337999999999999,"15min":20.108000000000001},"last":0.42699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2888241152,"data_stored":26089827630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4686966784,"data_stored":42577835281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1799536640,"data_stored":16491493347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:18:35.485 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:35.485 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:35.485 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:35.485 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:35.485 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:35.486 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:18:35.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.666+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:35.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.704+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:35.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.711+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:35.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.711+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:35.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.711+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:35.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.744+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:35.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.744+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:35.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.744+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:35.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.765+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:35.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:35.775+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:36.202 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:18:36.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:18:36.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:36.938+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:36.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:36.938+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:36.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:36.938+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:36.953 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:18:36.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2023-10-03T18:18:36.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:36.982+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:36.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:36.982+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:37.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:37.061+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:37.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:37.061+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:37.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:37.061+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:37.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:37.180+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:37.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:37.288+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:37.629 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.0 on osd.2 to deep-scrub 2023-10-03T18:18:37.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:18:38.361 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:18:38.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:18:39.064 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:18:39.083 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:18:39.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.648+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:39.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.648+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:39.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.755+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:39.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.768+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:39.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.768+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:39.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.777+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:39.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.777+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:39.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:39.777+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:42.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:42.904+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:42.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:42.957+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:42.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:42.957+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:43.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.011+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:43.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.011+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:43.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.095+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:43.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.095+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:43.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.095+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:43.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.113+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:43.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.148+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:43.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.148+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:18:43.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.149+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:43.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.149+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:43.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.187+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:43.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.187+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:43.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.187+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:43.201 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.201+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:43.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.516+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:43.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.516+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:43.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.539+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:43.562 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.560+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:43.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.578+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:43.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.578+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:43.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.591+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:43.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.596+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:43.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:43.600+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:44.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:44.856+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:44.857 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:44.856+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:44.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:44.899+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:44.923 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:44.921+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:45.034 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.034+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:45.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.773+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:45.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.773+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:45.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.777+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:45.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.834+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:45.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.834+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:45.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.834+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:45.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.834+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:45.839 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:45.834+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:54.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:54.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.811+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:54.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.842+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:54.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.842+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:54.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.885+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:54.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.949+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:54.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:54.955+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:56.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.147+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:56.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:56.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:56.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:56.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:56.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:56.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:56.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:56.173+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:58.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.678+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:18:58.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:18:58.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:18:58.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.795+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:18:58.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.843+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:18:58.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.868+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:18:58.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.868+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:18:58.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.883+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:18:58.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:58.926+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:18:59.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:18:59.002+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:18:59.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:18:59.876 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:18:59.878 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:18:59.899 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2023-10-03T18:18:59.086742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23213311793,"num_objects":53354,"num_object_clones":48088,"num_object_copies":106708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53354,"num_whiteouts":0,"num_read":1254112,"num_read_kb":4596317,"num_write":1156905,"num_write_kb":4617203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80519,"ondisk_log_size":80519,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14777628,"kb_used_data":9233092,"kb_used_omap":6399,"kb_used_meta":5538112,"kb_avail":268337892,"statfs":{"total":289910292480,"available":274778001408,"internally_reserved":0,"allocated":9454686208,"data_stored":85485250439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":5671026835},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,16,6,14,21,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23265280,"num_objects":27,"num_object_clones":27,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":6852,"num_read_kb":22671,"num_write":5547,"num_write_kb":22188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005753"},"pg_stats":[{"pgid":"2.7","version":"248'167199","reported_seq":303425,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:57.970325+0000","last_change":"2023-10-03T18:18:55.749904+0000","last_active":"2023-10-03T18:18:57.970325+0000","last_peered":"2023-10-03T18:18:57.970325+0000","last_clean":"2023-10-03T18:18:57.970325+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:18:57.970325+0000","last_undegraded":"2023-10-03T18:18:57.970325+0000","last_fullsized":"2023-10-03T18:18:57.970325+0000","mapping_epoch":12,"log_start":"223'157163","ondisk_log_start":"223'157163","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":196,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:29:05.694901+0000","scrub_duration":505.38444223300002,"objects_trimmed":242,"snaptrim_duration":48.379644198000001,"stat_sum":{"num_bytes":2892792292,"num_objects":9109,"num_object_clones":8458,"num_object_copies":18218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9109,"num_whiteouts":0,"num_read":153328,"num_read_kb":571722,"num_write":143884,"num_write_kb":574294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"4"}]},{"pgid":"2.6","version":"248'154791","reported_seq":281207,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:58.011456+0000","last_change":"2023-10-03T18:18:57.139222+0000","last_active":"2023-10-03T18:18:58.011456+0000","last_peered":"2023-10-03T18:18:58.011456+0000","last_clean":"2023-10-03T18:18:58.011456+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:18:58.011456+0000","last_undegraded":"2023-10-03T18:18:58.011456+0000","last_fullsized":"2023-10-03T18:18:58.011456+0000","mapping_epoch":12,"log_start":"220'144725","ondisk_log_start":"220'144725","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":113,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":248,"snaptrim_duration":49.77636115,"stat_sum":{"num_bytes":2671878868,"num_objects":6090,"num_object_clones":5484,"num_object_copies":12180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6090,"num_whiteouts":0,"num_read":142902,"num_read_kb":528851,"num_write":132831,"num_write_kb":530429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"7"}]},{"pgid":"2.5","version":"248'186086","reported_seq":334903,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:57.934886+0000","last_change":"2023-10-03T18:18:57.016726+0000","last_active":"2023-10-03T18:18:57.934886+0000","last_peered":"2023-10-03T18:18:57.934886+0000","last_clean":"2023-10-03T18:18:57.934886+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:18:57.934886+0000","last_undegraded":"2023-10-03T18:18:57.934886+0000","last_fullsized":"2023-10-03T18:18:57.934886+0000","mapping_epoch":12,"log_start":"229'176022","ondisk_log_start":"229'176022","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":135,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":214,"snaptrim_duration":46.429994596999997,"stat_sum":{"num_bytes":3133346445,"num_objects":7841,"num_object_clones":7131,"num_object_copies":15682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7841,"num_whiteouts":0,"num_read":173686,"num_read_kb":623850,"num_write":156429,"num_write_kb":623568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"5"}]},{"pgid":"2.0","version":"248'173481","reported_seq":315891,"reported_epoch":248,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:18:56.308748+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:18:56.308748+0000","last_peered":"2023-10-03T18:18:56.308748+0000","last_clean":"2023-10-03T18:18:56.308748+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:18:56.308748+0000","last_undegraded":"2023-10-03T18:18:56.308748+0000","last_fullsized":"2023-10-03T18:18:56.308748+0000","mapping_epoch":12,"log_start":"222'163469","ondisk_log_start":"222'163469","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5692,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":220,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 671s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3027063439,"num_objects":10339,"num_object_clones":9654,"num_object_copies":20678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10339,"num_whiteouts":0,"num_read":159599,"num_read_kb":595873,"num_write":150986,"num_write_kb":602774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":408,"reported_epoch":247,"state":"active+clean","last_fresh":"2023-10-03T18:18:06.329588+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:18:06.329588+0000","last_peered":"2023-10-03T18:18:06.329588+0000","last_clean":"2023-10-03T18:18:06.329588+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:18:06.329588+0000","last_undegraded":"2023-10-03T18:18:06.329588+0000","last_fullsized":"2023-10-03T18:18:06.329588+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"248'179184","reported_seq":322661,"reported_epoch":248,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:18:57.806009+0000","last_change":"2023-10-03T18:18:57.015652+0000","last_active":"2023-10-03T18:18:57.806009+0000","last_peered":"2023-10-03T18:18:57.806009+0000","last_clean":"2023-10-03T18:18:57.806009+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:18:57.806009+0000","last_undegraded":"2023-10-03T18:18:57.806009+0000","last_fullsized":"2023-10-03T18:18:57.806009+0000","mapping_epoch":12,"log_start":"229'169138","ondisk_log_start":"229'169138","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":127,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":258,"snaptrim_duration":49.672243315999999,"stat_sum":{"num_bytes":3079471844,"num_objects":7494,"num_object_clones":6797,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7494,"num_whiteouts":0,"num_read":162801,"num_read_kb":607297,"num_write":153277,"num_write_kb":611843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"}]},{"pgid":"2.1","version":"248'172439","reported_seq":302948,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-03T18:18:56.331944+0000","last_change":"2023-10-03T18:18:41.647367+0000","last_active":"2023-10-03T18:18:56.331944+0000","last_peered":"2023-10-03T18:18:56.331944+0000","last_clean":"2023-10-03T18:18:56.331944+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:18:56.331944+0000","last_undegraded":"2023-10-03T18:18:56.331944+0000","last_fullsized":"2023-10-03T18:18:56.331944+0000","mapping_epoch":12,"log_start":"230'162392","ondisk_log_start":"230'162392","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:03:59.704648+0000","scrub_duration":424.08306342600002,"objects_trimmed":513,"snaptrim_duration":34.311248571999997,"stat_sum":{"num_bytes":2805183088,"num_objects":2995,"num_object_clones":2360,"num_object_copies":5990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":152141,"num_read_kb":553656,"num_write":139604,"num_write_kb":556826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"}]},{"pgid":"2.2","version":"248'172481","reported_seq":307310,"reported_epoch":248,"state":"active+clean","last_fresh":"2023-10-03T18:18:57.203529+0000","last_change":"2023-10-03T18:18:42.718436+0000","last_active":"2023-10-03T18:18:57.203529+0000","last_peered":"2023-10-03T18:18:57.203529+0000","last_clean":"2023-10-03T18:18:57.203529+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:18:57.203529+0000","last_undegraded":"2023-10-03T18:18:57.203529+0000","last_fullsized":"2023-10-03T18:18:57.203529+0000","mapping_epoch":12,"log_start":"230'162417","ondisk_log_start":"230'162417","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:54:13.755597+0000","scrub_duration":370.574046178,"objects_trimmed":522,"snaptrim_duration":35.391956325999999,"stat_sum":{"num_bytes":2808484613,"num_objects":2996,"num_object_clones":2354,"num_object_copies":5992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2996,"num_whiteouts":0,"num_read":159688,"num_read_kb":563161,"num_write":140854,"num_write_kb":561230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"}]},{"pgid":"2.4","version":"248'163413","reported_seq":295469,"reported_epoch":248,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:18:57.822363+0000","last_change":"2023-10-03T18:18:57.139379+0000","last_active":"2023-10-03T18:18:57.822363+0000","last_peered":"2023-10-03T18:18:57.822363+0000","last_clean":"2023-10-03T18:18:57.822363+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:18:57.822363+0000","last_undegraded":"2023-10-03T18:18:57.822363+0000","last_fullsized":"2023-10-03T18:18:57.822363+0000","mapping_epoch":12,"log_start":"221'153328","ondisk_log_start":"221'153328","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":118,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":255,"snaptrim_duration":46.737811512,"stat_sum":{"num_bytes":2794500836,"num_objects":6488,"num_object_clones":5850,"num_object_copies":12976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6488,"num_whiteouts":0,"num_read":149861,"num_read_kb":551816,"num_write":138903,"num_write_kb":554345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23212721425,"num_objects":53352,"num_object_clones":48088,"num_object_copies":106704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53352,"num_whiteouts":0,"num_read":1254006,"num_read_kb":4596226,"num_write":1156768,"num_write_kb":4615309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9447522304,"data_stored":85468822050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80420,"ondisk_log_size":80420,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738923,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3404020,"kb_used_data":1772844,"kb_used_omap":969,"kb_used_meta":1630198,"kb_avail":90967820,"statfs":{"total":96636764160,"available":93151047680,"internally_reserved":0,"allocated":1815392256,"data_stored":16620836250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1669323630},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,13,0,1,4,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":1.5640000000000001,"5min":2.6960000000000002,"15min":2.5840000000000001},"min":{"1min":0.182,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":8.2469999999999999,"5min":18.321000000000002,"15min":32.173000000000002},"last":3.7570000000000001},{"interface":"front","average":{"1min":3.7810000000000001,"5min":2.5979999999999999,"15min":2.2149999999999999},"min":{"1min":0.23899999999999999,"5min":0.20899999999999999,"15min":0.17199999999999999},"max":{"1min":30.128,"5min":30.128,"15min":30.128},"last":0.40899999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:18:10 2023","interfaces":[{"interface":"back","average":{"1min":2.8799999999999999,"5min":2.4300000000000002,"15min":2.4470000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":17.274000000000001,"5min":17.274000000000001,"15min":18.768999999999998},"last":11.006},{"interface":"front","average":{"1min":1.508,"5min":1.8180000000000001,"15min":2.1680000000000001},"min":{"1min":0.29699999999999999,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":6.1719999999999997,"5min":8.7349999999999994,"15min":19.097000000000001},"last":6.1479999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738922,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4826492,"kb_used_data":2844404,"kb_used_omap":2419,"kb_used_meta":1979660,"kb_avail":89545348,"statfs":{"total":96636764160,"available":91694436352,"internally_reserved":0,"allocated":2912669696,"data_stored":26124909829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":2027172112},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,2,5,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.0800000000000001,"15min":3.3849999999999998},"min":{"1min":0.19800000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":10.66,"5min":31.474,"15min":31.474},"last":0.28799999999999998},{"interface":"front","average":{"1min":4.3419999999999996,"5min":3.5299999999999998,"15min":3.032},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":19.498999999999999,"5min":30.273,"15min":30.273},"last":5.4269999999999996}]},{"osd":2,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.996,"15min":1.8460000000000001},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":8.5449999999999999,"5min":13.209,"15min":13.209},"last":0.33400000000000002},{"interface":"front","average":{"1min":2.028,"5min":2.1459999999999999,"15min":2.0350000000000001},"min":{"1min":0.251,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.383,"5min":16.033999999999999,"15min":16.033999999999999},"last":12.151999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738903,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6547116,"kb_used_data":4615844,"kb_used_omap":3010,"kb_used_meta":1928253,"kb_avail":87824724,"statfs":{"total":96636764160,"available":89932517376,"internally_reserved":0,"allocated":4726624256,"data_stored":42739504360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":1974531093},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,4,8,16,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":3.8900000000000001,"5min":2.573,"15min":2.556},"min":{"1min":0.185,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":18.613,"5min":18.613,"15min":18.613},"last":3.927},{"interface":"front","average":{"1min":3.363,"5min":2.9119999999999999,"15min":2.722},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":21.498999999999999,"5min":21.498999999999999,"15min":21.498999999999999},"last":3.3879999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":1.6399999999999999,"15min":2.1749999999999998},"min":{"1min":0.221,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":8.3580000000000005,"5min":13.948,"15min":21.326000000000001},"last":4.2039999999999997},{"interface":"front","average":{"1min":2.7770000000000001,"5min":3.0310000000000001,"15min":2.6640000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":10.093999999999999,"5min":12.337999999999999,"15min":20.108000000000001},"last":3.4390000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2909982720,"data_stored":26119494958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4724731904,"data_stored":42730018065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1812807680,"data_stored":16619309027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:18:59.900 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:59.901 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:59.901 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:59.901 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:59.901 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:18:59.901 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:19:03.071 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.065+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:19:03.104 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:19:03.095+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:19:03.104 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:19:03.095+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:19:03.104 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:19:03.095+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:19:03.104 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:19:03.095+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:19:03.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:03.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.161+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:03.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.177+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:03.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.189+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:03.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.294+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:03.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.294+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:03.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.307+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:03.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.307+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:03.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.307+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:03.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:03.307+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:08.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.730+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:19:08.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.730+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:08.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.730+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:08.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.730+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:08.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.836+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:08.845 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.843+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:08.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.873+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:08.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.873+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:08.881 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.880+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:08.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.896+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:08.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:08.929+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:09.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:09.991+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:10.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.019+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:10.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.019+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:10.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.041+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:10.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.041+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:10.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.100+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:10.253 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.248+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:10.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.503+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:10.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:10.947+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:19:11.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.037+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:11.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.139+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:11.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.334+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:11.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.356+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:11.362 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.362+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:11.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.384+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:11.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.384+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:11.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.384+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:11.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.384+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:11.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:11.384+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:12.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.407+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:12.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.407+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:12.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.407+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:12.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.425+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:12.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.469+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:12.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.469+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:12.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.469+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:12.487 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.487+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:12.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:12.620+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:13.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:13.699+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:13.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:13.699+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:13.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:13.712+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:13.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:13.721+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:13.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:13.723+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:15.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:15.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:15.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:15.825+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:16.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.034+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:16.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.035+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:16.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.206+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:16.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.206+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:16.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.206+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:16.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.333+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:16.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.333+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:16.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.473+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:19:16.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.473+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:16.477 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.473+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:16.477 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.473+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:16.484 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.484+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:16.512 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.511+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:16.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.548+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:16.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.548+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:16.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.570+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:16.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.649+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:16.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:16.656+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:17.946 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:17.945+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:17.946 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:17.945+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:18.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.065+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:18.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.065+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:18.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.081+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:18.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.081+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:18.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.081+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:18.166 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.161+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:18.166 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.161+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:18.166 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.161+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:18.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.166+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:18.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.168+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:18.218 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.215+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:18.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.227+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:18.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.227+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:18.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.228+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:18.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.304+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:18.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:18.340+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:19.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.306+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:19.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.319+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:19.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.319+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:19.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.344+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:19.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.375+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:19.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.375+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:19.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.375+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:19.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.375+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:19.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.375+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:19.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.879+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:19.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.879+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:19.903 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:19:19.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.925+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:19.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.938+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:19.953 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:19.951+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:20.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:20.031+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:20.590 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:19:20.605 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:19:20.625 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1643,"stamp":"2023-10-03T18:19:19.096319+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23233120049,"num_objects":54703,"num_object_clones":49437,"num_object_copies":109406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54703,"num_whiteouts":0,"num_read":1261780,"num_read_kb":4617217,"num_write":1161901,"num_write_kb":4636467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80610,"ondisk_log_size":80610,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15036196,"kb_used_data":9272704,"kb_used_omap":6399,"kb_used_meta":5757056,"kb_avail":268079324,"statfs":{"total":289910292480,"available":274513227776,"internally_reserved":0,"allocated":9495248896,"data_stored":88065323617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":5895225491},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,6,9,9,8,12,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10915738,"num_objects":1261,"num_object_clones":1263,"num_object_copies":2522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1261,"num_whiteouts":0,"num_read":3522,"num_read_kb":10999,"num_write":2729,"num_write_kb":10608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.807171"},"pg_stats":[{"pgid":"2.7","version":"255'168230","reported_seq":304797,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:14.844973+0000","last_change":"2023-10-03T18:19:12.766549+0000","last_active":"2023-10-03T18:19:14.844973+0000","last_peered":"2023-10-03T18:19:14.844973+0000","last_clean":"2023-10-03T18:19:14.844973+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:19:14.844973+0000","last_undegraded":"2023-10-03T18:19:14.844973+0000","last_fullsized":"2023-10-03T18:19:14.844973+0000","mapping_epoch":12,"log_start":"229'158168","ondisk_log_start":"229'158168","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":214,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:46:02.528045+0000","scrub_duration":505.38444223300002,"objects_trimmed":16,"snaptrim_duration":0.023728583000000001,"stat_sum":{"num_bytes":2895127114,"num_objects":9318,"num_object_clones":8665,"num_object_copies":18636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9318,"num_whiteouts":0,"num_read":154175,"num_read_kb":574178,"num_write":144478,"num_write_kb":576592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"6"}]},{"pgid":"2.6","version":"255'155695","reported_seq":282496,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:14.759587+0000","last_change":"2023-10-03T18:19:13.525851+0000","last_active":"2023-10-03T18:19:14.759587+0000","last_peered":"2023-10-03T18:19:14.759587+0000","last_clean":"2023-10-03T18:19:14.759587+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:19:14.759587+0000","last_undegraded":"2023-10-03T18:19:14.759587+0000","last_fullsized":"2023-10-03T18:19:14.759587+0000","mapping_epoch":12,"log_start":"228'145630","ondisk_log_start":"228'145630","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":131,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":1.5335206110000001,"stat_sum":{"num_bytes":2673980065,"num_objects":6271,"num_object_clones":5666,"num_object_copies":12542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6271,"num_whiteouts":0,"num_read":143645,"num_read_kb":531110,"num_write":133359,"num_write_kb":532487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"9"}]},{"pgid":"2.5","version":"255'187117","reported_seq":336328,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:14.445409+0000","last_change":"2023-10-03T18:19:13.520350+0000","last_active":"2023-10-03T18:19:14.445409+0000","last_peered":"2023-10-03T18:19:14.445409+0000","last_clean":"2023-10-03T18:19:14.445409+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:19:14.445409+0000","last_undegraded":"2023-10-03T18:19:14.445409+0000","last_fullsized":"2023-10-03T18:19:14.445409+0000","mapping_epoch":12,"log_start":"231'177027","ondisk_log_start":"231'177027","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":152,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":6.2147093059999996,"stat_sum":{"num_bytes":3135693352,"num_objects":8005,"num_object_clones":7297,"num_object_copies":16010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8005,"num_whiteouts":0,"num_read":174769,"num_read_kb":626322,"num_write":157032,"num_write_kb":625842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"8"}]},{"pgid":"2.0","version":"255'174670","reported_seq":317550,"reported_epoch":255,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:19:16.577782+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:19:16.577782+0000","last_peered":"2023-10-03T18:19:16.577782+0000","last_clean":"2023-10-03T18:19:16.577782+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:19:16.577782+0000","last_undegraded":"2023-10-03T18:19:16.577782+0000","last_fullsized":"2023-10-03T18:19:16.577782+0000","mapping_epoch":12,"log_start":"229'164570","ondisk_log_start":"229'164570","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":5840,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":240,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 692s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3030065910,"num_objects":10649,"num_object_clones":9962,"num_object_copies":21298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10649,"num_whiteouts":0,"num_read":160577,"num_read_kb":598860,"num_write":151727,"num_write_kb":605667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":415,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-03T18:19:06.517330+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:19:06.517330+0000","last_peered":"2023-10-03T18:19:06.517330+0000","last_clean":"2023-10-03T18:19:06.517330+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:19:06.517330+0000","last_undegraded":"2023-10-03T18:19:06.517330+0000","last_fullsized":"2023-10-03T18:19:06.517330+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"255'180185","reported_seq":324059,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:14.377867+0000","last_change":"2023-10-03T18:19:10.689148+0000","last_active":"2023-10-03T18:19:14.377867+0000","last_peered":"2023-10-03T18:19:14.377867+0000","last_clean":"2023-10-03T18:19:14.377867+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:19:14.377867+0000","last_undegraded":"2023-10-03T18:19:14.377867+0000","last_fullsized":"2023-10-03T18:19:14.377867+0000","mapping_epoch":12,"log_start":"231'170144","ondisk_log_start":"231'170144","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":145,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":23,"snaptrim_duration":3.3673942179999998,"stat_sum":{"num_bytes":3081921252,"num_objects":7691,"num_object_clones":6994,"num_object_copies":15382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7691,"num_whiteouts":0,"num_read":163680,"num_read_kb":609884,"num_write":153878,"num_write_kb":614182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"2"}]},{"pgid":"2.1","version":"255'173838","reported_seq":304780,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:16.577872+0000","last_change":"2023-10-03T18:19:13.857601+0000","last_active":"2023-10-03T18:19:16.577872+0000","last_peered":"2023-10-03T18:19:16.577872+0000","last_clean":"2023-10-03T18:19:16.577872+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:19:16.577872+0000","last_undegraded":"2023-10-03T18:19:16.577872+0000","last_fullsized":"2023-10-03T18:19:16.577872+0000","mapping_epoch":12,"log_start":"231'163796","ondisk_log_start":"231'163796","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:06.944935+0000","scrub_duration":424.08306342600002,"objects_trimmed":135,"snaptrim_duration":6.5505356599999995,"stat_sum":{"num_bytes":2807923260,"num_objects":3041,"num_object_clones":2407,"num_object_copies":6082,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3041,"num_whiteouts":0,"num_read":153215,"num_read_kb":556560,"num_write":140306,"num_write_kb":559529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"6"}]},{"pgid":"2.2","version":"255'173894","reported_seq":309170,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:17.421871+0000","last_change":"2023-10-03T18:19:17.308171+0000","last_active":"2023-10-03T18:19:17.421871+0000","last_peered":"2023-10-03T18:19:17.421871+0000","last_clean":"2023-10-03T18:19:17.421871+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:19:17.421871+0000","last_undegraded":"2023-10-03T18:19:17.421871+0000","last_fullsized":"2023-10-03T18:19:17.421871+0000","mapping_epoch":12,"log_start":"231'163822","ondisk_log_start":"231'163822","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:36:59.106541+0000","scrub_duration":370.574046178,"objects_trimmed":134,"snaptrim_duration":9.9781495239999991,"stat_sum":{"num_bytes":2811192170,"num_objects":3078,"num_object_clones":2434,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":160899,"num_read_kb":566035,"num_write":141540,"num_write_kb":563838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"7"}]},{"pgid":"2.4","version":"255'164370","reported_seq":296813,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:14.453305+0000","last_change":"2023-10-03T18:19:12.730641+0000","last_active":"2023-10-03T18:19:14.453305+0000","last_peered":"2023-10-03T18:19:14.453305+0000","last_clean":"2023-10-03T18:19:14.453305+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:19:14.453305+0000","last_undegraded":"2023-10-03T18:19:14.453305+0000","last_fullsized":"2023-10-03T18:19:14.453305+0000","mapping_epoch":12,"log_start":"229'154331","ondisk_log_start":"229'154331","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":135,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":37,"snaptrim_duration":5.4250211139999998,"stat_sum":{"num_bytes":2796626558,"num_objects":6648,"num_object_clones":6012,"num_object_copies":13296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6648,"num_whiteouts":0,"num_read":150714,"num_read_kb":554177,"num_write":139444,"num_write_kb":556436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23232529681,"num_objects":54701,"num_object_clones":49437,"num_object_copies":109402,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54701,"num_whiteouts":0,"num_read":1261674,"num_read_kb":4617126,"num_write":1161764,"num_write_kb":4634573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9487523840,"data_stored":88218331682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80511,"ondisk_log_size":80511,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738927,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3481592,"kb_used_data":1781424,"kb_used_omap":969,"kb_used_meta":1699190,"kb_avail":90890248,"statfs":{"total":96636764160,"available":93071613952,"internally_reserved":0,"allocated":1824178176,"data_stored":17179105643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1739971438},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,2,2,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.3119999999999998,"5min":2.7360000000000002,"15min":2.6219999999999999},"min":{"1min":0.193,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":9.0429999999999993,"5min":18.321000000000002,"15min":32.173000000000002},"last":2.1160000000000001},{"interface":"front","average":{"1min":2.9950000000000001,"5min":2.7789999999999999,"15min":2.3330000000000002},"min":{"1min":0.27300000000000002,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":10.586,"5min":30.128,"15min":30.128},"last":0.309}]},{"osd":1,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0659999999999998,"5min":2.5510000000000002,"15min":2.5950000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":11.006,"5min":17.274000000000001,"15min":18.768999999999998},"last":4.2300000000000004},{"interface":"front","average":{"1min":3.3620000000000001,"5min":2.0910000000000002,"15min":2.3290000000000002},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":12.346,"5min":12.346,"15min":19.097000000000001},"last":1.8029999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738926,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4925828,"kb_used_data":2857656,"kb_used_omap":2419,"kb_used_meta":2065740,"kb_avail":89446012,"statfs":{"total":96636764160,"available":91592716288,"internally_reserved":0,"allocated":2926239744,"data_stored":26998190192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":2115318032},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5299999999999998,"5min":3.0800000000000001,"15min":3.3849999999999998},"min":{"1min":0.19800000000000001,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":10.66,"5min":31.474,"15min":31.474},"last":0.215},{"interface":"front","average":{"1min":4.3419999999999996,"5min":3.5299999999999998,"15min":3.032},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":19.498999999999999,"5min":30.273,"15min":30.273},"last":7.2960000000000003}]},{"osd":2,"last update":"Tue Oct 3 18:18:30 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.996,"15min":1.8460000000000001},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":8.5449999999999999,"5min":13.209,"15min":13.209},"last":1.992},{"interface":"front","average":{"1min":2.028,"5min":2.1459999999999999,"15min":2.0350000000000001},"min":{"1min":0.251,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.383,"5min":16.033999999999999,"15min":16.033999999999999},"last":10.537000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738906,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6628776,"kb_used_data":4633624,"kb_used_omap":3010,"kb_used_meta":1992125,"kb_avail":87743064,"statfs":{"total":96636764160,"available":89848897536,"internally_reserved":0,"allocated":4744830976,"data_stored":43888027782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":2039936021},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,4,6,6,10,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":3.8900000000000001,"5min":2.573,"15min":2.556},"min":{"1min":0.185,"5min":0.183,"15min":0.16300000000000001},"max":{"1min":18.613,"5min":18.613,"15min":18.613},"last":2.4489999999999998},{"interface":"front","average":{"1min":3.363,"5min":2.9119999999999999,"15min":2.722},"min":{"1min":0.19900000000000001,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":21.498999999999999,"5min":21.498999999999999,"15min":21.498999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Tue Oct 3 18:18:18 2023","interfaces":[{"interface":"back","average":{"1min":2.0270000000000001,"5min":1.6399999999999999,"15min":2.1749999999999998},"min":{"1min":0.221,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":8.3580000000000005,"5min":13.948,"15min":21.326000000000001},"last":3.524},{"interface":"front","average":{"1min":2.7770000000000001,"5min":3.0310000000000001,"15min":2.6640000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.20699999999999999},"max":{"1min":10.093999999999999,"5min":12.337999999999999,"15min":20.108000000000001},"last":1.361}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2923352064,"data_stored":27036024059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4742639616,"data_stored":43998811409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1821532160,"data_stored":17183496214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:19:20.627 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:20.627 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:20.627 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:20.627 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:20.628 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:20.628 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:19:24.781 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:24.777+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:24.787 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:24.786+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:19:24.788 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:24.786+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:28.133 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:28.122+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:28.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:28.277+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:28.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:28.277+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:31.548 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:31.546+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:31.560 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:31.557+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:34.321 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:34.317+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:34.331 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:34.330+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:19:34.380 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:34.375+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:38.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.306+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:38.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.325+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:38.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.366+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:38.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.366+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:38.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.366+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:38.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.366+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:38.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.374+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:38.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.381+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:38.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.381+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:19:38.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.381+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:38.383 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.381+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:38.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.382+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:38.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.382+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:38.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.433+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:38.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.433+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:38.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.433+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:38.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.433+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:38.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.449+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:38.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:38.449+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:39.690 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:39.686+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:39.690 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:39.686+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:19:39.690 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:39.686+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:39.742 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:39.740+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:39.744 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:39.743+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:39.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:39.758+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:39.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:39.758+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:39.975 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:39.974+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:40.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:40.140+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:40.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:40.140+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:40.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:40.166+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:40.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:40.170+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:40.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:40.218+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:40.629 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:19:41.368 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:19:41.380 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:19:41.394 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1654,"stamp":"2023-10-03T18:19:41.100675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23256774449,"num_objects":55580,"num_object_clones":50314,"num_object_copies":111160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55580,"num_whiteouts":0,"num_read":1267404,"num_read_kb":4639585,"num_write":1167573,"num_write_kb":4659155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80391,"ondisk_log_size":80391,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15351408,"kb_used_data":9316696,"kb_used_omap":6399,"kb_used_meta":6028288,"kb_avail":267764112,"statfs":{"total":289910292480,"available":274190450688,"internally_reserved":0,"allocated":9540296704,"data_stored":89934996065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6553453,"internal_metadata":6172967059},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3,14,15,12,23,20,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12546048,"num_objects":171,"num_object_clones":171,"num_object_copies":342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":171,"num_whiteouts":0,"num_read":2931,"num_read_kb":11676,"num_write":3006,"num_write_kb":12024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002316"},"pg_stats":[{"pgid":"2.7","version":"255'169402","reported_seq":306392,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:36.940251+0000","last_change":"2023-10-03T18:19:35.303554+0000","last_active":"2023-10-03T18:19:36.940251+0000","last_peered":"2023-10-03T18:19:36.940251+0000","last_clean":"2023-10-03T18:19:36.940251+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:19:36.940251+0000","last_undegraded":"2023-10-03T18:19:36.940251+0000","last_fullsized":"2023-10-03T18:19:36.940251+0000","mapping_epoch":12,"log_start":"231'159373","ondisk_log_start":"231'159373","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":210,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:46:02.528045+0000","scrub_duration":505.38444223300002,"objects_trimmed":124,"snaptrim_duration":27.992623657999999,"stat_sum":{"num_bytes":2897965642,"num_objects":9459,"num_object_clones":8806,"num_object_copies":18918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9459,"num_whiteouts":0,"num_read":154877,"num_read_kb":576962,"num_write":145171,"num_write_kb":579364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"}]},{"pgid":"2.6","version":"255'156759","reported_seq":283996,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:36.913263+0000","last_change":"2023-10-03T18:19:33.110259+0000","last_active":"2023-10-03T18:19:36.913263+0000","last_peered":"2023-10-03T18:19:36.913263+0000","last_clean":"2023-10-03T18:19:36.913263+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:19:36.913263+0000","last_undegraded":"2023-10-03T18:19:36.913263+0000","last_fullsized":"2023-10-03T18:19:36.913263+0000","mapping_epoch":12,"log_start":"230'146736","ondisk_log_start":"230'146736","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":127,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":129,"snaptrim_duration":25.76888945,"stat_sum":{"num_bytes":2676642465,"num_objects":6401,"num_object_clones":5796,"num_object_copies":12802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6401,"num_whiteouts":0,"num_read":144332,"num_read_kb":533842,"num_write":133992,"num_write_kb":535019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"3"}]},{"pgid":"2.5","version":"255'188461","reported_seq":338113,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:36.924374+0000","last_change":"2023-10-03T18:19:35.505500+0000","last_active":"2023-10-03T18:19:36.924374+0000","last_peered":"2023-10-03T18:19:36.924374+0000","last_clean":"2023-10-03T18:19:36.924374+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:19:36.924374+0000","last_undegraded":"2023-10-03T18:19:36.924374+0000","last_fullsized":"2023-10-03T18:19:36.924374+0000","mapping_epoch":12,"log_start":"231'178433","ondisk_log_start":"231'178433","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":149,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":136,"snaptrim_duration":28.199857621,"stat_sum":{"num_bytes":3139093032,"num_objects":8203,"num_object_clones":7495,"num_object_copies":16406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8203,"num_whiteouts":0,"num_read":175533,"num_read_kb":629350,"num_write":157832,"num_write_kb":629042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"}]},{"pgid":"2.0","version":"255'175803","reported_seq":319129,"reported_epoch":255,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:19:36.699562+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:19:36.699562+0000","last_peered":"2023-10-03T18:19:36.699562+0000","last_clean":"2023-10-03T18:19:36.699562+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:19:36.699562+0000","last_undegraded":"2023-10-03T18:19:36.699562+0000","last_fullsized":"2023-10-03T18:19:36.699562+0000","mapping_epoch":12,"log_start":"231'165772","ondisk_log_start":"231'165772","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":6040,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":240,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 712s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3032920822,"num_objects":10900,"num_object_clones":10213,"num_object_copies":21800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10900,"num_whiteouts":0,"num_read":161244,"num_read_kb":601524,"num_write":152409,"num_write_kb":608395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":415,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-03T18:19:06.517330+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:19:06.517330+0000","last_peered":"2023-10-03T18:19:06.517330+0000","last_clean":"2023-10-03T18:19:06.517330+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:19:06.517330+0000","last_undegraded":"2023-10-03T18:19:06.517330+0000","last_fullsized":"2023-10-03T18:19:06.517330+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"255'181536","reported_seq":325874,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:36.868543+0000","last_change":"2023-10-03T18:19:35.527989+0000","last_active":"2023-10-03T18:19:36.868543+0000","last_peered":"2023-10-03T18:19:36.868543+0000","last_clean":"2023-10-03T18:19:36.868543+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:19:36.868543+0000","last_undegraded":"2023-10-03T18:19:36.868543+0000","last_fullsized":"2023-10-03T18:19:36.868543+0000","mapping_epoch":12,"log_start":"231'171453","ondisk_log_start":"231'171453","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":142,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":159,"snaptrim_duration":24.394226937999999,"stat_sum":{"num_bytes":3085099748,"num_objects":7844,"num_object_clones":7147,"num_object_copies":15688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7844,"num_whiteouts":0,"num_read":164478,"num_read_kb":613064,"num_write":154654,"num_write_kb":617286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"}]},{"pgid":"2.1","version":"255'175374","reported_seq":306591,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:36.666765+0000","last_change":"2023-10-03T18:19:35.639742+0000","last_active":"2023-10-03T18:19:36.666765+0000","last_peered":"2023-10-03T18:19:36.666765+0000","last_clean":"2023-10-03T18:19:36.666765+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:19:36.666765+0000","last_undegraded":"2023-10-03T18:19:36.666765+0000","last_fullsized":"2023-10-03T18:19:36.666765+0000","mapping_epoch":12,"log_start":"239'165306","ondisk_log_start":"239'165306","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:06.944935+0000","scrub_duration":424.08306342600002,"objects_trimmed":467,"snaptrim_duration":28.335115979000001,"stat_sum":{"num_bytes":2810470972,"num_objects":2943,"num_object_clones":2309,"num_object_copies":5886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2943,"num_whiteouts":0,"num_read":153829,"num_read_kb":559000,"num_write":140929,"num_write_kb":562021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"}]},{"pgid":"2.2","version":"255'175439","reported_seq":311033,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:37.578206+0000","last_change":"2023-10-03T18:19:36.943864+0000","last_active":"2023-10-03T18:19:37.578206+0000","last_peered":"2023-10-03T18:19:37.578206+0000","last_clean":"2023-10-03T18:19:37.578206+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:19:37.578206+0000","last_undegraded":"2023-10-03T18:19:37.578206+0000","last_fullsized":"2023-10-03T18:19:37.578206+0000","mapping_epoch":12,"log_start":"241'165437","ondisk_log_start":"241'165437","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:36:59.106541+0000","scrub_duration":370.574046178,"objects_trimmed":450,"snaptrim_duration":29.620772172999999,"stat_sum":{"num_bytes":2814042986,"num_objects":2994,"num_object_clones":2350,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":161553,"num_read_kb":568639,"num_write":142209,"num_write_kb":566514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"}]},{"pgid":"2.4","version":"255'165664","reported_seq":298534,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:36.934383+0000","last_change":"2023-10-03T18:19:35.313893+0000","last_active":"2023-10-03T18:19:36.934383+0000","last_peered":"2023-10-03T18:19:36.934383+0000","last_clean":"2023-10-03T18:19:36.934383+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:19:36.934383+0000","last_undegraded":"2023-10-03T18:19:36.934383+0000","last_fullsized":"2023-10-03T18:19:36.934383+0000","mapping_epoch":12,"log_start":"231'155636","ondisk_log_start":"231'155636","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":132,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":137,"snaptrim_duration":23.513590306000001,"stat_sum":{"num_bytes":2799948414,"num_objects":6834,"num_object_clones":6198,"num_object_copies":13668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":151452,"num_read_kb":557113,"num_write":140240,"num_write_kb":559620,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"8"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23256184081,"num_objects":55578,"num_object_clones":50314,"num_object_copies":111156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55578,"num_whiteouts":0,"num_read":1267298,"num_read_kb":4639494,"num_write":1167436,"num_write_kb":4657261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9533431808,"data_stored":89919056418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64690,"internal_metadata":0},"log_size":80292,"ondisk_log_size":80292,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738931,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3537796,"kb_used_data":1789052,"kb_used_omap":969,"kb_used_meta":1747766,"kb_avail":90834044,"statfs":{"total":96636764160,"available":93014061056,"internally_reserved":0,"allocated":1831989248,"data_stored":17471531371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1789713262},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,8,2,3,10,17,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.3119999999999998,"5min":2.7360000000000002,"15min":2.6219999999999999},"min":{"1min":0.193,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":9.0429999999999993,"5min":18.321000000000002,"15min":32.173000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.9950000000000001,"5min":2.7789999999999999,"15min":2.3330000000000002},"min":{"1min":0.27300000000000002,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":10.586,"5min":30.128,"15min":30.128},"last":0.23799999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0659999999999998,"5min":2.5510000000000002,"15min":2.5950000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":11.006,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.27100000000000002},{"interface":"front","average":{"1min":3.3620000000000001,"5min":2.0910000000000002,"15min":2.3290000000000002},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":12.346,"5min":12.346,"15min":19.097000000000001},"last":0.32000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738930,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4977920,"kb_used_data":2870904,"kb_used_omap":2419,"kb_used_meta":2104588,"kb_avail":89393920,"statfs":{"total":96636764160,"available":91539374080,"internally_reserved":0,"allocated":2939805696,"data_stored":27500851312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2477808,"internal_metadata":2155098384},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":3.2360000000000002,"5min":3.1190000000000002,"15min":3.2989999999999999},"min":{"1min":0.184,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":19.969999999999999,"5min":31.474,"15min":31.474},"last":0.51200000000000001},{"interface":"front","average":{"1min":3.0489999999999999,"5min":3.423,"15min":3.0470000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":8.1340000000000003,"5min":30.273,"15min":30.273},"last":1.653}]},{"osd":2,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":2.7559999999999998,"5min":2.1419999999999999,"15min":1.873},"min":{"1min":0.27300000000000002,"5min":0.192,"15min":0.182},"max":{"1min":13.058,"5min":13.058,"15min":13.209},"last":0.77500000000000002},{"interface":"front","average":{"1min":2.931,"5min":2.0710000000000002,"15min":2.1000000000000001},"min":{"1min":0.247,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":12.151999999999999,"5min":12.151999999999999,"15min":16.033999999999999},"last":0.247}]}]},{"osd":1,"up_from":8,"seq":34359738910,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6835692,"kb_used_data":4656740,"kb_used_omap":3010,"kb_used_meta":2175933,"kb_avail":87536148,"statfs":{"total":96636764160,"available":89637015552,"internally_reserved":0,"allocated":4768501760,"data_stored":44962613382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":2228155413},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,6,10,7,12,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":4.3899999999999997,"5min":3.0409999999999999,"15min":2.718},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":25.145,"5min":25.145,"15min":25.145},"last":2.972},{"interface":"front","average":{"1min":2.7749999999999999,"5min":2.9609999999999999,"15min":2.637},"min":{"1min":0.27400000000000002,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":16.061,"5min":21.498999999999999,"15min":21.498999999999999},"last":2.3610000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.982,"15min":2.1579999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":9.5069999999999997,"5min":13.948,"15min":21.326000000000001},"last":2.3279999999999998},{"interface":"front","average":{"1min":2.8599999999999999,"5min":3.036,"15min":2.6680000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":9.4870000000000001,"5min":12.337999999999999,"15min":20.108000000000001},"last":5.8959999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2937180160,"data_stored":27499330811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22298,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4766507008,"data_stored":44959610129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1829744640,"data_stored":17460115478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:19:41.395 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:41.396 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:41.396 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:41.396 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:41.396 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:19:41.397 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:19:45.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.718+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:45.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:45.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:45.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.718+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:45.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.801+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:45.834 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.831+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:45.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.831+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:45.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.893+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:45.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.893+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:45.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:45.898+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:47.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.329+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:47.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.374+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:47.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.374+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:47.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.390+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:47.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.390+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:47.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.457+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:47.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.457+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.457+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.457+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.457+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:47.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.576+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:47.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.602+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:47.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.602+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:47.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.602+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:47.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.602+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:47.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.633+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:47.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.635+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:47.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.635+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:47.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.635+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:47.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:47.635+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:48.724 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:48.724+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:48.775 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:48.775+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:19:48.788 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:48.785+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:49.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.648+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:49.669 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.664+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:49.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.781+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:49.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.781+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:49.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.781+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:49.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.804+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:49.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:49.818+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:50.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.421+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:50.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.427+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:50.437 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.427+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:50.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.501+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:50.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.520+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:50.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:50.550+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:51.751 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.751+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:51.766 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.765+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:51.766 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.765+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:51.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.777+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:51.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.878+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:51.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.878+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:51.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.878+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:51.879 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.878+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:51.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:51.879+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:53.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.767+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:53.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.783+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:53.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.783+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:53.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.783+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:53.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.783+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:53.790 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.790+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:53.790 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:53.790+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:55.198 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:55.198+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:19:55.222 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:55.222+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:19:55.222 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:19:55.222+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:19:55.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:55.705+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:55.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:55.748+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:56.284 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.276+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:19:56.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.276+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:19:56.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.276+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:19:56.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.276+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:56.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.276+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:56.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.282+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:56.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.308+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:56.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.308+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:56.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:56.328+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:57.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.851+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:57.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.851+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:57.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.865+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:57.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.865+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:57.866 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.865+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:57.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.906+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:57.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:57.917+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:19:59.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.518+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:19:59.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.570+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:19:59.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.570+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:19:59.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.570+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:19:59.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.602+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:19:59.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.614+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:19:59.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:19:59.614+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:01.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:20:02.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.045+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:02.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.046+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:02.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.046+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:02.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.046+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:02.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.090+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:02.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.090+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:02.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.090+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:02.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.114+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:02.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.147+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:02.202 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:20:02.217 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:20:02.227 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1665,"stamp":"2023-10-03T18:20:01.227764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23286884145,"num_objects":55607,"num_object_clones":50341,"num_object_copies":111214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55607,"num_whiteouts":0,"num_read":1275878,"num_read_kb":4669366,"num_write":1174805,"num_write_kb":4688083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80582,"ondisk_log_size":80582,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15535516,"kb_used_data":9374400,"kb_used_omap":6091,"kb_used_meta":6154996,"kb_avail":267580004,"statfs":{"total":289910292480,"available":274001924096,"internally_reserved":0,"allocated":9599385600,"data_stored":90166092385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6237194,"internal_metadata":6302716918},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,7,17,12,18,28,12],"upper_bound":512},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15687680,"num_objects":-71,"num_object_clones":-71,"num_object_copies":-142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-71,"num_whiteouts":0,"num_read":5024,"num_read_kb":16029,"num_write":3801,"num_write_kb":15204,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.120442"},"pg_stats":[{"pgid":"2.7","version":"255'170640","reported_seq":308419,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:57.664143+0000","last_change":"2023-10-03T18:19:56.359860+0000","last_active":"2023-10-03T18:19:57.664143+0000","last_peered":"2023-10-03T18:19:57.664143+0000","last_clean":"2023-10-03T18:19:57.664143+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:19:57.664143+0000","last_undegraded":"2023-10-03T18:19:57.664143+0000","last_fullsized":"2023-10-03T18:19:57.664143+0000","mapping_epoch":12,"log_start":"234'160577","ondisk_log_start":"234'160577","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":206,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:46:02.528045+0000","scrub_duration":505.38444223300002,"objects_trimmed":248,"snaptrim_duration":49.048929223999998,"stat_sum":{"num_bytes":2901549642,"num_objects":9419,"num_object_clones":8766,"num_object_copies":18838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9419,"num_whiteouts":0,"num_read":155886,"num_read_kb":580608,"num_write":146047,"num_write_kb":582868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"4"}]},{"pgid":"2.6","version":"255'157999","reported_seq":285991,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:57.650293+0000","last_change":"2023-10-03T18:19:54.653358+0000","last_active":"2023-10-03T18:19:57.650293+0000","last_peered":"2023-10-03T18:19:57.650293+0000","last_clean":"2023-10-03T18:19:57.650293+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:19:57.650293+0000","last_undegraded":"2023-10-03T18:19:57.650293+0000","last_fullsized":"2023-10-03T18:19:57.650293+0000","mapping_epoch":12,"log_start":"231'147938","ondisk_log_start":"231'147938","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":123,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":226,"snaptrim_duration":47.311997929,"stat_sum":{"num_bytes":2680570529,"num_objects":6406,"num_object_clones":5801,"num_object_copies":12812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6406,"num_whiteouts":0,"num_read":145315,"num_read_kb":537359,"num_write":134921,"num_write_kb":538735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"}]},{"pgid":"2.5","version":"255'189902","reported_seq":340469,"reported_epoch":255,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:19:57.629805+0000","last_change":"2023-10-03T18:19:56.890793+0000","last_active":"2023-10-03T18:19:57.629805+0000","last_peered":"2023-10-03T18:19:57.629805+0000","last_clean":"2023-10-03T18:19:57.629805+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:19:57.629805+0000","last_undegraded":"2023-10-03T18:19:57.629805+0000","last_fullsized":"2023-10-03T18:19:57.629805+0000","mapping_epoch":12,"log_start":"241'179835","ondisk_log_start":"241'179835","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":145,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":271,"snaptrim_duration":49.585145599000001,"stat_sum":{"num_bytes":3143168552,"num_objects":8184,"num_object_clones":7476,"num_object_copies":16368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8184,"num_whiteouts":0,"num_read":176803,"num_read_kb":633654,"num_write":158827,"num_write_kb":633022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"2"}]},{"pgid":"2.0","version":"255'176956","reported_seq":321168,"reported_epoch":255,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-03T18:19:56.828547+0000","last_change":"2023-10-03T18:07:44.413121+0000","last_active":"2023-10-03T18:19:56.828547+0000","last_peered":"2023-10-03T18:19:56.828547+0000","last_clean":"2023-10-03T18:19:56.828547+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:19:56.828547+0000","last_undegraded":"2023-10-03T18:19:56.828547+0000","last_fullsized":"2023-10-03T18:19:56.828547+0000","mapping_epoch":12,"log_start":"233'166872","ondisk_log_start":"233'166872","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":6213,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":240,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 732s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":34.447302012000002,"stat_sum":{"num_bytes":3036799734,"num_objects":10989,"num_object_clones":10302,"num_object_copies":21978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10989,"num_whiteouts":0,"num_read":162300,"num_read_kb":605358,"num_write":153296,"num_write_kb":611943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"88'99","reported_seq":416,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-03T18:19:07.407134+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:19:07.407134+0000","last_peered":"2023-10-03T18:19:07.407134+0000","last_clean":"2023-10-03T18:19:07.407134+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:19:07.407134+0000","last_undegraded":"2023-10-03T18:19:07.407134+0000","last_fullsized":"2023-10-03T18:19:07.407134+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"255'182854","reported_seq":327990,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:57.642159+0000","last_change":"2023-10-03T18:19:56.924299+0000","last_active":"2023-10-03T18:19:57.642159+0000","last_peered":"2023-10-03T18:19:57.642159+0000","last_clean":"2023-10-03T18:19:57.642159+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:19:57.642159+0000","last_undegraded":"2023-10-03T18:19:57.642159+0000","last_fullsized":"2023-10-03T18:19:57.642159+0000","mapping_epoch":12,"log_start":"241'172853","ondisk_log_start":"241'172853","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":138,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":277,"snaptrim_duration":47.176190108999997,"stat_sum":{"num_bytes":3088986852,"num_objects":7829,"num_object_clones":7132,"num_object_copies":15658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7829,"num_whiteouts":0,"num_read":165516,"num_read_kb":616801,"num_write":155603,"num_write_kb":621082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"2"}]},{"pgid":"2.1","version":"255'176464","reported_seq":308467,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-03T18:19:56.826991+0000","last_change":"2023-10-03T18:19:40.387300+0000","last_active":"2023-10-03T18:19:56.826991+0000","last_peered":"2023-10-03T18:19:56.826991+0000","last_clean":"2023-10-03T18:19:56.826991+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:19:56.826991+0000","last_undegraded":"2023-10-03T18:19:56.826991+0000","last_fullsized":"2023-10-03T18:19:56.826991+0000","mapping_epoch":12,"log_start":"241'166407","ondisk_log_start":"241'166407","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10057,"log_dups_size":0,"ondisk_log_size":10057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:06.944935+0000","scrub_duration":424.08306342600002,"objects_trimmed":522,"snaptrim_duration":33.085616125999998,"stat_sum":{"num_bytes":2814063164,"num_objects":2988,"num_object_clones":2354,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":154821,"num_read_kb":562574,"num_write":141791,"num_write_kb":565469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"}]},{"pgid":"2.2","version":"255'176529","reported_seq":312933,"reported_epoch":255,"state":"active+clean","last_fresh":"2023-10-03T18:19:57.650265+0000","last_change":"2023-10-03T18:19:41.967072+0000","last_active":"2023-10-03T18:19:57.650265+0000","last_peered":"2023-10-03T18:19:57.650265+0000","last_clean":"2023-10-03T18:19:57.650265+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:19:57.650265+0000","last_undegraded":"2023-10-03T18:19:57.650265+0000","last_fullsized":"2023-10-03T18:19:57.650265+0000","mapping_epoch":12,"log_start":"241'166439","ondisk_log_start":"241'166439","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:36:59.106541+0000","scrub_duration":370.574046178,"objects_trimmed":525,"snaptrim_duration":34.644392938999999,"stat_sum":{"num_bytes":2817487722,"num_objects":3002,"num_object_clones":2358,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":162663,"num_read_kb":572219,"num_write":143050,"num_write_kb":569878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"}]},{"pgid":"2.4","version":"255'166898","reported_seq":300568,"reported_epoch":255,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:19:57.650281+0000","last_change":"2023-10-03T18:19:56.361714+0000","last_active":"2023-10-03T18:19:57.650281+0000","last_peered":"2023-10-03T18:19:57.650281+0000","last_clean":"2023-10-03T18:19:57.650281+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:19:57.650281+0000","last_undegraded":"2023-10-03T18:19:57.650281+0000","last_fullsized":"2023-10-03T18:19:57.650281+0000","mapping_epoch":12,"log_start":"232'156838","ondisk_log_start":"232'156838","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":128,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":256,"snaptrim_duration":45.905362253,"stat_sum":{"num_bytes":2803667582,"num_objects":6788,"num_object_clones":6152,"num_object_copies":13576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6788,"num_whiteouts":0,"num_read":152468,"num_read_kb":560702,"num_write":141133,"num_write_kb":563192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"be","length":"a"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23286293777,"num_objects":55605,"num_object_clones":50341,"num_object_copies":111210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55605,"num_whiteouts":0,"num_read":1275772,"num_read_kb":4669275,"num_write":1174668,"num_write_kb":4686189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":10705,"num_omap_keys":132,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9592487936,"data_stored":90159524386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":62914,"internal_metadata":0},"log_size":80483,"ondisk_log_size":80483,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738935,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3558156,"kb_used_data":1799684,"kb_used_omap":969,"kb_used_meta":1757494,"kb_avail":90813684,"statfs":{"total":96636764160,"available":92993212416,"internally_reserved":0,"allocated":1842876416,"data_stored":17599134059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1799674734},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,3,9,18,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.3119999999999998,"5min":2.7360000000000002,"15min":2.6219999999999999},"min":{"1min":0.193,"5min":0.182,"15min":0.16600000000000001},"max":{"1min":9.0429999999999993,"5min":18.321000000000002,"15min":32.173000000000002},"last":4.8840000000000003},{"interface":"front","average":{"1min":2.9950000000000001,"5min":2.7789999999999999,"15min":2.3330000000000002},"min":{"1min":0.27300000000000002,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":10.586,"5min":30.128,"15min":30.128},"last":0.52400000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":3.0659999999999998,"5min":2.5510000000000002,"15min":2.5950000000000002},"min":{"1min":0.23599999999999999,"5min":0.23499999999999999,"15min":0.20399999999999999},"max":{"1min":11.006,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":3.3620000000000001,"5min":2.0910000000000002,"15min":2.3290000000000002},"min":{"1min":0.27900000000000003,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":12.346,"5min":12.346,"15min":19.097000000000001},"last":0.27000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738934,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4994804,"kb_used_data":2889128,"kb_used_omap":2110,"kb_used_meta":2103553,"kb_avail":89377036,"statfs":{"total":96636764160,"available":91522084864,"internally_reserved":0,"allocated":2958467072,"data_stored":27480580208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2161549,"internal_metadata":2154038387},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":3.2360000000000002,"5min":3.1190000000000002,"15min":3.2989999999999999},"min":{"1min":0.184,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":19.969999999999999,"5min":31.474,"15min":31.474},"last":4.2009999999999996},{"interface":"front","average":{"1min":3.0489999999999999,"5min":3.423,"15min":3.0470000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":8.1340000000000003,"5min":30.273,"15min":30.273},"last":0.26100000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":2.7559999999999998,"5min":2.1419999999999999,"15min":1.873},"min":{"1min":0.27300000000000002,"5min":0.192,"15min":0.182},"max":{"1min":13.058,"5min":13.058,"15min":13.209},"last":1.2310000000000001},{"interface":"front","average":{"1min":2.931,"5min":2.0710000000000002,"15min":2.1000000000000001},"min":{"1min":0.247,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":12.151999999999999,"5min":12.151999999999999,"15min":16.033999999999999},"last":1.9359999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738914,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6982556,"kb_used_data":4685588,"kb_used_omap":3010,"kb_used_meta":2293949,"kb_avail":87389284,"statfs":{"total":96636764160,"available":89486626816,"internally_reserved":0,"allocated":4798042112,"data_stored":45086378118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3083243,"internal_metadata":2349003797},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,5,3,7,9,9,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":4.3899999999999997,"5min":3.0409999999999999,"15min":2.718},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":25.145,"5min":25.145,"15min":25.145},"last":0.37},{"interface":"front","average":{"1min":2.7749999999999999,"5min":2.9609999999999999,"15min":2.637},"min":{"1min":0.27400000000000002,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":16.061,"5min":21.498999999999999,"15min":21.498999999999999},"last":2.5289999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.982,"15min":2.1579999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":9.5069999999999997,"5min":13.948,"15min":21.326000000000001},"last":5.702},{"interface":"front","average":{"1min":2.8599999999999999,"5min":3.036,"15min":2.6680000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":9.4870000000000001,"5min":12.337999999999999,"15min":20.108000000000001},"last":2.4969999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2955313152,"data_stored":27480104187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20522,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4796506112,"data_stored":45078238481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35385,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1840668672,"data_stored":17601181718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:20:02.229 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:02.229 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:02.230 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:02.230 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:02.230 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:02.230 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:20:02.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.270+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:02.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.270+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:02.272 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.270+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:02.272 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.270+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:02.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.344+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:02.388 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.382+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:02.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:02.440+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:03.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:03.549+0000 7fc515f98700 -1 reset not still connected to 0x562209741790 2023-10-03T18:20:03.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:03.549+0000 7fc515f98700 -1 reset not still connected to 0x562215002c30 2023-10-03T18:20:03.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:03.609+0000 7fc515f98700 -1 reset not still connected to 0x56222efd7c70 2023-10-03T18:20:03.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:03.735+0000 7fc515f98700 -1 reset not still connected to 0x56223718cf70 2023-10-03T18:20:04.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:04.540+0000 7fc515f98700 -1 reset not still connected to 0x562214744d00 2023-10-03T18:20:04.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:04.541+0000 7fc515f98700 -1 reset not still connected to 0x562215002c30 2023-10-03T18:20:04.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:04.561+0000 7fc515f98700 -1 reset not still connected to 0x5622272ad1e0 2023-10-03T18:20:04.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:04.561+0000 7fc515f98700 -1 reset not still connected to 0x56222efd7c70 2023-10-03T18:20:04.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:04.600+0000 7fc515f98700 -1 reset not still connected to 0x56223718cf70 2023-10-03T18:20:06.836 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:20:06.833+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:20:06.836 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:20:06.833+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:20:06.836 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:20:06.833+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:20:06.836 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:20:06.833+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:20:10.313 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.313+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:10.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.358+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:10.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.394+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:10.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.394+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:10.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.429+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:10.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.447+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:10.533 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.532+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:10.533 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.532+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:10.543 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.543+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:10.543 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.543+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:10.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:10.614+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:11.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.166+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:11.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.168+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:11.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.168+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:11.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.197+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:11.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.317+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:11.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.317+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:11.377 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.377+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:11.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.535+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:11.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:11.587+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:13.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.465+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:13.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.508+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:13.509 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.508+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:13.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.512+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:13.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.547+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:13.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.687+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:13.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.687+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:13.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.810+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:13.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.852+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:13.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:13.925+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:14.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.275+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:14.334 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.333+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:14.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.333+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:14.339 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.338+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:14.339 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.339+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:14.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:14.445+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:15.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:15.168+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:15.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:15.172+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:18.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.227+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:18.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.249+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:18.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.342+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:18.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.480+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:18.505 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.501+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:18.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.514+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:18.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.514+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:18.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.514+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:18.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.680+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:18.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:18.684+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:20.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.253+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:20.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.265+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:20.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.280+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:20.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.367+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:20.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.616+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:20.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.670+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:20.694 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.693+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:20.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.693+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:20.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.713+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:20.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.754+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:20.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:20.783+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:22.231 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:20:23.018 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:20:23.024 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:20:23.056 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2023-10-03T18:20:22.155882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23310792497,"num_objects":57017,"num_object_clones":51751,"num_object_copies":114034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57017,"num_whiteouts":0,"num_read":1284516,"num_read_kb":4693588,"num_write":1180806,"num_write_kb":4711367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80481,"ondisk_log_size":80481,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14059400,"kb_used_data":9420260,"kb_used_omap":6100,"kb_used_meta":4633003,"kb_avail":269056120,"statfs":{"total":289910292480,"available":275513466880,"internally_reserved":0,"allocated":9646346240,"data_stored":92905831321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6246424,"internal_metadata":4744196072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,6,14,31,19,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9547776,"num_objects":874,"num_object_clones":874,"num_object_copies":1748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":874,"num_whiteouts":0,"num_read":2131,"num_read_kb":8446,"num_write":2322,"num_write_kb":9277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.026217"},"pg_stats":[{"pgid":"2.7","version":"266'171943","reported_seq":310115,"reported_epoch":266,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:19.741830+0000","last_change":"2023-10-03T18:20:17.071985+0000","last_active":"2023-10-03T18:20:19.741830+0000","last_peered":"2023-10-03T18:20:19.741830+0000","last_clean":"2023-10-03T18:20:19.741830+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:20:19.741830+0000","last_undegraded":"2023-10-03T18:20:19.741830+0000","last_fullsized":"2023-10-03T18:20:19.741830+0000","mapping_epoch":12,"log_start":"241'161883","ondisk_log_start":"241'161883","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":223,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:18.883531+0000","scrub_duration":505.38444223300002,"objects_trimmed":26,"snaptrim_duration":2.1121412290000001,"stat_sum":{"num_bytes":2904461746,"num_objects":9639,"num_object_clones":8989,"num_object_copies":19278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9639,"num_whiteouts":0,"num_read":156856,"num_read_kb":583516,"num_write":146775,"num_write_kb":585703,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"}]},{"pgid":"2.6","version":"266'159173","reported_seq":287585,"reported_epoch":266,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:19.595468+0000","last_change":"2023-10-03T18:20:16.301411+0000","last_active":"2023-10-03T18:20:19.595468+0000","last_peered":"2023-10-03T18:20:19.595468+0000","last_clean":"2023-10-03T18:20:19.595468+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:20:19.595468+0000","last_undegraded":"2023-10-03T18:20:19.595468+0000","last_fullsized":"2023-10-03T18:20:19.595468+0000","mapping_epoch":12,"log_start":"241'149142","ondisk_log_start":"241'149142","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10031,"log_dups_size":0,"ondisk_log_size":10031,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.1190038820000003,"stat_sum":{"num_bytes":2683261499,"num_objects":6586,"num_object_clones":5983,"num_object_copies":13172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6586,"num_whiteouts":0,"num_read":146145,"num_read_kb":539909,"num_write":135588,"num_write_kb":541359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"}]},{"pgid":"2.5","version":"266'191265","reported_seq":342324,"reported_epoch":266,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:19.673968+0000","last_change":"2023-10-03T18:20:17.051307+0000","last_active":"2023-10-03T18:20:19.673968+0000","last_peered":"2023-10-03T18:20:19.673968+0000","last_clean":"2023-10-03T18:20:19.673968+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:20:19.673968+0000","last_undegraded":"2023-10-03T18:20:19.673968+0000","last_fullsized":"2023-10-03T18:20:19.673968+0000","mapping_epoch":12,"log_start":"241'181238","ondisk_log_start":"241'181238","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":162,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.8690222389999995,"stat_sum":{"num_bytes":3146392105,"num_objects":8454,"num_object_clones":7746,"num_object_copies":16908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8454,"num_whiteouts":0,"num_read":178097,"num_read_kb":636892,"num_write":159639,"num_write_kb":636140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"}]},{"pgid":"2.0","version":"266'178868","reported_seq":323478,"reported_epoch":266,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:21.833826+0000","last_change":"2023-10-03T18:20:20.837256+0000","last_active":"2023-10-03T18:20:21.833826+0000","last_peered":"2023-10-03T18:20:21.833826+0000","last_clean":"2023-10-03T18:20:21.833826+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:20:21.833826+0000","last_undegraded":"2023-10-03T18:20:21.833826+0000","last_fullsized":"2023-10-03T18:20:21.833826+0000","mapping_epoch":12,"log_start":"241'168784","ondisk_log_start":"241'168784","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":250,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:35:05.078733+0000","scrub_duration":734.24538164700004,"objects_trimmed":63,"snaptrim_duration":3.6371816269999999,"stat_sum":{"num_bytes":3040301965,"num_objects":11051,"num_object_clones":10361,"num_object_copies":22102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11051,"num_whiteouts":0,"num_read":163400,"num_read_kb":608789,"num_write":154176,"num_write_kb":615375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c4","length":"4"}]},{"pgid":"1.0","version":"88'99","reported_seq":427,"reported_epoch":265,"state":"active+clean","last_fresh":"2023-10-03T18:20:12.297109+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:20:12.297109+0000","last_peered":"2023-10-03T18:20:12.297109+0000","last_clean":"2023-10-03T18:20:12.297109+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:20:12.297109+0000","last_undegraded":"2023-10-03T18:20:12.297109+0000","last_fullsized":"2023-10-03T18:20:12.297109+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"266'184127","reported_seq":329664,"reported_epoch":266,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:19.648427+0000","last_change":"2023-10-03T18:20:18.681586+0000","last_active":"2023-10-03T18:20:19.648427+0000","last_peered":"2023-10-03T18:20:19.648427+0000","last_clean":"2023-10-03T18:20:19.648427+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:20:19.648427+0000","last_undegraded":"2023-10-03T18:20:19.648427+0000","last_fullsized":"2023-10-03T18:20:19.648427+0000","mapping_epoch":12,"log_start":"241'174057","ondisk_log_start":"241'174057","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":155,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":3.702065535,"stat_sum":{"num_bytes":3091948362,"num_objects":8083,"num_object_clones":7384,"num_object_copies":16166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8083,"num_whiteouts":0,"num_read":166481,"num_read_kb":619703,"num_write":156348,"num_write_kb":623991,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"5"}]},{"pgid":"2.1","version":"266'178177","reported_seq":310612,"reported_epoch":266,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:21.848702+0000","last_change":"2023-10-03T18:20:21.377122+0000","last_active":"2023-10-03T18:20:21.848702+0000","last_peered":"2023-10-03T18:20:21.848702+0000","last_clean":"2023-10-03T18:20:21.848702+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:20:21.848702+0000","last_undegraded":"2023-10-03T18:20:21.848702+0000","last_fullsized":"2023-10-03T18:20:21.848702+0000","mapping_epoch":12,"log_start":"241'168114","ondisk_log_start":"241'168114","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:17:50.359035+0000","scrub_duration":424.08306342600002,"objects_trimmed":70,"snaptrim_duration":4.2155001729999997,"stat_sum":{"num_bytes":2817282671,"num_objects":3074,"num_object_clones":2439,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":156063,"num_read_kb":565921,"num_write":142609,"num_write_kb":568637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"9"}]},{"pgid":"2.2","version":"266'177881","reported_seq":314701,"reported_epoch":266,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:17.818123+0000","last_change":"2023-10-03T18:20:16.300482+0000","last_active":"2023-10-03T18:20:17.818123+0000","last_peered":"2023-10-03T18:20:17.818123+0000","last_clean":"2023-10-03T18:20:17.818123+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:20:17.818123+0000","last_undegraded":"2023-10-03T18:20:17.818123+0000","last_fullsized":"2023-10-03T18:20:17.818123+0000","mapping_epoch":12,"log_start":"241'167848","ondisk_log_start":"241'167848","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:05:56.801509+0000","scrub_duration":370.574046178,"objects_trimmed":6,"snaptrim_duration":4.101576197,"stat_sum":{"num_bytes":2820039379,"num_objects":3111,"num_object_clones":2470,"num_object_copies":6222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3111,"num_whiteouts":0,"num_read":163867,"num_read_kb":575026,"num_write":143699,"num_write_kb":572339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"6"}]},{"pgid":"2.4","version":"266'168156","reported_seq":302299,"reported_epoch":266,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:19.642718+0000","last_change":"2023-10-03T18:20:18.599216+0000","last_active":"2023-10-03T18:20:19.642718+0000","last_peered":"2023-10-03T18:20:19.642718+0000","last_clean":"2023-10-03T18:20:19.642718+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:20:19.642718+0000","last_undegraded":"2023-10-03T18:20:19.642718+0000","last_fullsized":"2023-10-03T18:20:19.642718+0000","mapping_epoch":12,"log_start":"241'158142","ondisk_log_start":"241'158142","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":144,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":1.3914713380000001,"stat_sum":{"num_bytes":2806514402,"num_objects":7017,"num_object_clones":6379,"num_object_copies":14034,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7017,"num_whiteouts":0,"num_read":153501,"num_read_kb":563741,"num_write":141835,"num_write_kb":565929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"c4","length":"4"},{"start":"d2","length":"7"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23310202129,"num_objects":57015,"num_object_clones":51751,"num_object_copies":114030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57015,"num_whiteouts":0,"num_read":1284410,"num_read_kb":4693497,"num_write":1180669,"num_write_kb":4709473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9638629376,"data_stored":93007125026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72144,"internal_metadata":0},"log_size":80382,"ondisk_log_size":80382,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738940,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3636392,"kb_used_data":1809888,"kb_used_omap":969,"kb_used_meta":1825526,"kb_avail":90735448,"statfs":{"total":96636764160,"available":92913098752,"internally_reserved":0,"allocated":1853325312,"data_stored":18058937091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1869339502},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,6,14,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.4039999999999999,"15min":2.5720000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":7.8440000000000003,"5min":13.859999999999999,"15min":32.173000000000002},"last":1.508},{"interface":"front","average":{"1min":1.3680000000000001,"5min":2.4049999999999998,"15min":2.2210000000000001},"min":{"1min":0.21299999999999999,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":6.7149999999999999,"5min":30.128,"15min":30.128},"last":5.4290000000000003}]},{"osd":1,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":2.153,"15min":2.5819999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":4.4610000000000003,"5min":17.274000000000001,"15min":18.768999999999998},"last":10.628},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.0030000000000001,"15min":2.3719999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":7.5549999999999997,"5min":12.346,"15min":19.097000000000001},"last":10.815}]}]},{"osd":0,"up_from":8,"seq":34359738938,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3763308,"kb_used_data":2901856,"kb_used_omap":2116,"kb_used_meta":859323,"kb_avail":90608532,"statfs":{"total":96636764160,"available":92783136768,"internally_reserved":0,"allocated":2971500544,"data_stored":28389738408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":879947263},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,8],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":3.2360000000000002,"5min":3.1190000000000002,"15min":3.2989999999999999},"min":{"1min":0.184,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":19.969999999999999,"5min":31.474,"15min":31.474},"last":0.26600000000000001},{"interface":"front","average":{"1min":3.0489999999999999,"5min":3.423,"15min":3.0470000000000002},"min":{"1min":0.23000000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":8.1340000000000003,"5min":30.273,"15min":30.273},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:19:32 2023","interfaces":[{"interface":"back","average":{"1min":2.7559999999999998,"5min":2.1419999999999999,"15min":1.873},"min":{"1min":0.27300000000000002,"5min":0.192,"15min":0.182},"max":{"1min":13.058,"5min":13.058,"15min":13.209},"last":0.745},{"interface":"front","average":{"1min":2.931,"5min":2.0710000000000002,"15min":2.1000000000000001},"min":{"1min":0.247,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":12.151999999999999,"5min":12.151999999999999,"15min":16.033999999999999},"last":0.66100000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738918,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6659700,"kb_used_data":4708516,"kb_used_omap":3014,"kb_used_meta":1948153,"kb_avail":87712140,"statfs":{"total":96636764160,"available":89817231360,"internally_reserved":0,"allocated":4821520384,"data_stored":46457155822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3086725,"internal_metadata":1994909307},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,3,7,16,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":4.3899999999999997,"5min":3.0409999999999999,"15min":2.718},"min":{"1min":0.182,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":25.145,"5min":25.145,"15min":25.145},"last":7.5289999999999999},{"interface":"front","average":{"1min":2.7749999999999999,"5min":2.9609999999999999,"15min":2.637},"min":{"1min":0.27400000000000002,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":16.061,"5min":21.498999999999999,"15min":21.498999999999999},"last":7.399}]},{"osd":2,"last update":"Tue Oct 3 18:19:24 2023","interfaces":[{"interface":"back","average":{"1min":2.9100000000000001,"5min":1.982,"15min":2.1579999999999999},"min":{"1min":0.22700000000000001,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":9.5069999999999997,"5min":13.948,"15min":21.326000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":2.8599999999999999,"5min":3.036,"15min":2.6680000000000001},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":9.4870000000000001,"5min":12.337999999999999,"15min":20.108000000000001},"last":7.3259999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2968334336,"data_stored":28403993547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4819460096,"data_stored":46532056337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38867,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1850834944,"data_stored":18071075142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:20:23.058 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:23.058 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:23.059 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:23.059 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:23.059 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:20:23.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.379+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:23.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.500+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:23.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.634+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:23.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.635+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:23.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.637+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:23.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.650+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:23.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.691+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:23.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.691+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:23.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.691+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:23.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.706+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:23.837 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.836+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:23.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:23.894+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:24.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.052+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:24.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.060+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:24.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.066+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:24.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.066+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:24.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.111+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:24.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.111+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:24.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.161+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:24.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.161+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:24.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.237+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:24.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.639+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:24.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.650+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:24.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.650+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:24.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.695+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:24.763 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.761+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:24.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.802+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:24.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.802+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:24.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:24.925+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:25.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:25.087+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:25.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:25.162+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:27.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.286+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:27.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.369+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:27.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.396+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:27.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.396+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:27.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.396+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:27.486 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.485+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:27.495 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:27.493+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:30.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.754+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:30.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.787+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:30.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.802+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:30.809 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.807+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:30.809 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.807+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:30.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.946+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:30.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:30.971+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:31.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.027+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:31.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.052+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:31.140 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.137+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:31.227 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:20:31.215+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:20:31.242 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.240+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:31.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.863+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:31.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.863+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:31.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.948+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:31.982 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:31.980+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:32.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.105+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:32.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.105+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:32.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.105+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:32.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.123+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:32.209 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.202+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:32.209 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.202+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:32.209 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.202+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:32.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.955+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:32.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:32.990+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:33.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.036+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:33.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.042+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:33.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:33.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:33.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:33.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:33.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:33.054 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:33.048+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:34.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.010+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:34.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.064+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:34.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.159+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:34.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.166+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:34.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.166+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:34.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:34.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:34.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:34.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:34.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:34.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.168+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:34.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.172+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:34.178 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.176+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:34.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.288+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:34.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.556+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:34.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:34.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:34.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:34.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:34.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:34.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:34.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:34.564+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:35.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.189+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:35.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.256+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:35.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.256+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:35.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.289+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:35.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.343+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:35.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:35.391+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:39.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.302+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:39.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.305+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:39.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.310+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:39.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.368+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:39.406 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.405+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:39.406 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.405+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:39.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.434+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:39.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.459+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:39.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.603+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:39.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.653+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:39.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.672+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:39.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.893+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:39.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.940+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:39.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.940+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:39.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.945+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:39.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.978+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:39.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.978+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:39.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.978+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:39.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.980+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:39.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:39.980+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:40.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.198+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:40.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.294+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:40.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.334+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:40.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.350+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:40.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.350+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:40.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.350+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:40.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.351+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:40.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.400+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:40.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:40.400+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:42.500 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.489+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:42.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.552+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:42.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.552+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:42.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.622+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:42.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.638+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:42.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:42.769+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:43.060 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:20:43.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.644+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:43.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:43.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.678+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:43.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.679+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:43.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:43.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:43.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:43.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:43.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:43.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:43.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:43.680+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:43.754 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:20:43.760 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:20:43.777 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1701,"stamp":"2023-10-03T18:20:43.145692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23335003953,"num_objects":57223,"num_object_clones":51957,"num_object_copies":114446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57223,"num_whiteouts":0,"num_read":1290055,"num_read_kb":4715580,"num_write":1186685,"num_write_kb":4734883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80436,"ondisk_log_size":80436,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14431680,"kb_used_data":9470308,"kb_used_omap":6100,"kb_used_meta":4955243,"kb_avail":268683840,"statfs":{"total":289910292480,"available":275132252160,"internally_reserved":0,"allocated":9697595392,"data_stored":93853966697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6246424,"internal_metadata":5074169832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,2,5,11,5,13,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12406784,"num_objects":-111,"num_object_clones":-111,"num_object_copies":-222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-111,"num_whiteouts":0,"num_read":2794,"num_read_kb":11104,"num_write":3001,"num_write_kb":12004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005864"},"pg_stats":[{"pgid":"2.7","version":"270'173131","reported_seq":311799,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:41.531540+0000","last_change":"2023-10-03T18:20:40.418774+0000","last_active":"2023-10-03T18:20:41.531540+0000","last_peered":"2023-10-03T18:20:41.531540+0000","last_clean":"2023-10-03T18:20:41.531540+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:20:41.531540+0000","last_undegraded":"2023-10-03T18:20:41.531540+0000","last_fullsized":"2023-10-03T18:20:41.531540+0000","mapping_epoch":12,"log_start":"241'163091","ondisk_log_start":"241'163091","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":219,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:18.883531+0000","scrub_duration":505.38444223300002,"objects_trimmed":16,"snaptrim_duration":0.043460828999999999,"stat_sum":{"num_bytes":2907521458,"num_objects":9757,"num_object_clones":9107,"num_object_copies":19514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9757,"num_whiteouts":0,"num_read":157589,"num_read_kb":586436,"num_write":147522,"num_write_kb":588691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"1"}]},{"pgid":"2.6","version":"270'160333","reported_seq":289128,"reported_epoch":270,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:41.464995+0000","last_change":"2023-10-03T18:20:41.103077+0000","last_active":"2023-10-03T18:20:41.464995+0000","last_peered":"2023-10-03T18:20:41.464995+0000","last_clean":"2023-10-03T18:20:41.464995+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:20:41.464995+0000","last_undegraded":"2023-10-03T18:20:41.464995+0000","last_fullsized":"2023-10-03T18:20:41.464995+0000","mapping_epoch":12,"log_start":"241'150244","ondisk_log_start":"241'150244","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":136,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":3.9124662140000002,"stat_sum":{"num_bytes":2686202427,"num_objects":6679,"num_object_clones":6076,"num_object_copies":13358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6679,"num_whiteouts":0,"num_read":146763,"num_read_kb":542373,"num_write":136300,"num_write_kb":544207,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"4"}]},{"pgid":"2.5","version":"270'192548","reported_seq":344095,"reported_epoch":270,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:41.433924+0000","last_change":"2023-10-03T18:20:39.619037+0000","last_active":"2023-10-03T18:20:41.433924+0000","last_peered":"2023-10-03T18:20:41.433924+0000","last_clean":"2023-10-03T18:20:41.433924+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:20:41.433924+0000","last_undegraded":"2023-10-03T18:20:41.433924+0000","last_fullsized":"2023-10-03T18:20:41.433924+0000","mapping_epoch":12,"log_start":"242'182541","ondisk_log_start":"242'182541","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":159,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":2.428449729,"stat_sum":{"num_bytes":3149623849,"num_objects":8592,"num_object_clones":7884,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8592,"num_whiteouts":0,"num_read":178837,"num_read_kb":639836,"num_write":160421,"num_write_kb":639268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"1"}]},{"pgid":"2.0","version":"270'180353","reported_seq":325382,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:41.902822+0000","last_change":"2023-10-03T18:20:39.691019+0000","last_active":"2023-10-03T18:20:41.902822+0000","last_peered":"2023-10-03T18:20:41.902822+0000","last_clean":"2023-10-03T18:20:41.902822+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:20:41.902822+0000","last_undegraded":"2023-10-03T18:20:41.902822+0000","last_fullsized":"2023-10-03T18:20:41.902822+0000","mapping_epoch":12,"log_start":"241'170300","ondisk_log_start":"241'170300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":242,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:35:05.078733+0000","scrub_duration":734.24538164700004,"objects_trimmed":77,"snaptrim_duration":2.4756783439999999,"stat_sum":{"num_bytes":3043095437,"num_objects":10944,"num_object_clones":10254,"num_object_copies":21888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10944,"num_whiteouts":0,"num_read":164099,"num_read_kb":611553,"num_write":154860,"num_write_kb":618111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":431,"reported_epoch":269,"state":"active+clean","last_fresh":"2023-10-03T18:20:32.301393+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:20:32.301393+0000","last_peered":"2023-10-03T18:20:32.301393+0000","last_clean":"2023-10-03T18:20:32.301393+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:20:32.301393+0000","last_undegraded":"2023-10-03T18:20:32.301393+0000","last_fullsized":"2023-10-03T18:20:32.301393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"270'185372","reported_seq":331371,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:41.393511+0000","last_change":"2023-10-03T18:20:41.171714+0000","last_active":"2023-10-03T18:20:41.393511+0000","last_peered":"2023-10-03T18:20:41.393511+0000","last_clean":"2023-10-03T18:20:41.393511+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:20:41.393511+0000","last_undegraded":"2023-10-03T18:20:41.393511+0000","last_fullsized":"2023-10-03T18:20:41.393511+0000","mapping_epoch":12,"log_start":"241'175362","ondisk_log_start":"241'175362","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":152,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":1.041165275,"stat_sum":{"num_bytes":3095089994,"num_objects":8223,"num_object_clones":7524,"num_object_copies":16446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8223,"num_whiteouts":0,"num_read":167208,"num_read_kb":622591,"num_write":157115,"num_write_kb":627059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"8"}]},{"pgid":"2.1","version":"270'179629","reported_seq":312390,"reported_epoch":270,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:41.914806+0000","last_change":"2023-10-03T18:20:40.628698+0000","last_active":"2023-10-03T18:20:41.914806+0000","last_peered":"2023-10-03T18:20:41.914806+0000","last_clean":"2023-10-03T18:20:41.914806+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:20:41.914806+0000","last_undegraded":"2023-10-03T18:20:41.914806+0000","last_fullsized":"2023-10-03T18:20:41.914806+0000","mapping_epoch":12,"log_start":"248'169624","ondisk_log_start":"248'169624","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:17:50.359035+0000","scrub_duration":424.08306342600002,"objects_trimmed":75,"snaptrim_duration":3.4171502810000001,"stat_sum":{"num_bytes":2819859055,"num_objects":2920,"num_object_clones":2285,"num_object_copies":5840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2920,"num_whiteouts":0,"num_read":156659,"num_read_kb":568277,"num_write":143235,"num_write_kb":571141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"}]},{"pgid":"2.2","version":"271'179711","reported_seq":316987,"reported_epoch":271,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:20:43.099064+0000","last_change":"2023-10-03T18:20:42.883390+0000","last_active":"2023-10-03T18:20:43.099064+0000","last_peered":"2023-10-03T18:20:43.099064+0000","last_clean":"2023-10-03T18:20:43.099064+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:20:43.099064+0000","last_undegraded":"2023-10-03T18:20:43.099064+0000","last_fullsized":"2023-10-03T18:20:43.099064+0000","mapping_epoch":12,"log_start":"248'169655","ondisk_log_start":"248'169655","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:05:56.801509+0000","scrub_duration":370.574046178,"objects_trimmed":17,"snaptrim_duration":0.67092326899999999,"stat_sum":{"num_bytes":2823578323,"num_objects":3010,"num_object_clones":2369,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":164674,"num_read_kb":578222,"num_write":144558,"num_write_kb":575775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d8","length":"1"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"}]},{"pgid":"2.4","version":"270'169323","reported_seq":303861,"reported_epoch":270,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:20:41.419616+0000","last_change":"2023-10-03T18:20:40.411766+0000","last_active":"2023-10-03T18:20:41.419616+0000","last_peered":"2023-10-03T18:20:41.419616+0000","last_clean":"2023-10-03T18:20:41.419616+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:20:41.419616+0000","last_undegraded":"2023-10-03T18:20:41.419616+0000","last_fullsized":"2023-10-03T18:20:41.419616+0000","mapping_epoch":12,"log_start":"241'159246","ondisk_log_start":"241'159246","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":140,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":3.2202126999999998,"stat_sum":{"num_bytes":2809443042,"num_objects":7096,"num_object_clones":6458,"num_object_copies":14192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7096,"num_whiteouts":0,"num_read":154120,"num_read_kb":566201,"num_write":142537,"num_write_kb":568737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d7","length":"2"},{"start":"db","length":"3"},{"start":"e6","length":"a"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23334413585,"num_objects":57221,"num_object_clones":51957,"num_object_copies":114442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57221,"num_whiteouts":0,"num_read":1289949,"num_read_kb":4715489,"num_write":1186548,"num_write_kb":4732989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9688727552,"data_stored":93833075234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72144,"internal_metadata":0},"log_size":80337,"ondisk_log_size":80337,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738944,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3706944,"kb_used_data":1817976,"kb_used_omap":969,"kb_used_meta":1887990,"kb_avail":90664896,"statfs":{"total":96636764160,"available":92840853504,"internally_reserved":0,"allocated":1861607424,"data_stored":18022253459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1933302638},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.4039999999999999,"15min":2.5720000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":7.8440000000000003,"5min":13.859999999999999,"15min":32.173000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":1.3680000000000001,"5min":2.4049999999999998,"15min":2.2210000000000001},"min":{"1min":0.21299999999999999,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":6.7149999999999999,"5min":30.128,"15min":30.128},"last":1.4099999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":2.153,"15min":2.5819999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":4.4610000000000003,"5min":17.274000000000001,"15min":18.768999999999998},"last":1.988},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.0030000000000001,"15min":2.3719999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":7.5549999999999997,"5min":12.346,"15min":19.097000000000001},"last":3.4460000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738943,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3878216,"kb_used_data":2919164,"kb_used_omap":2116,"kb_used_meta":956923,"kb_avail":90493624,"statfs":{"total":96636764160,"available":92665470976,"internally_reserved":0,"allocated":2989223936,"data_stored":28904537176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":979889663},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,4,1,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.7370000000000001,"15min":3.169},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":12.895,"5min":19.969999999999999,"15min":31.474},"last":0.21099999999999999},{"interface":"front","average":{"1min":2.9900000000000002,"5min":3.0899999999999999,"15min":2.96},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":9.1750000000000007,"5min":19.498999999999999,"15min":30.273},"last":7.6639999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":2.0819999999999999,"15min":1.841},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":5.484,"5min":13.058,"15min":13.209},"last":1.869},{"interface":"front","average":{"1min":1.952,"5min":2.0910000000000002,"15min":2.04},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.1849999999999996,"5min":12.151999999999999,"15min":16.033999999999999},"last":0.22700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738922,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6846520,"kb_used_data":4733168,"kb_used_omap":3014,"kb_used_meta":2110329,"kb_avail":87525320,"statfs":{"total":96636764160,"available":89625927680,"internally_reserved":0,"allocated":4846764032,"data_stored":46927176062,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3086725,"internal_metadata":2160977531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,0,9,3,10,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":4.4219999999999997,"5min":3.4809999999999999,"15min":2.8450000000000002},"min":{"1min":0.25,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":19.361999999999998,"5min":25.145,"15min":25.145},"last":12.17},{"interface":"front","average":{"1min":3.9340000000000002,"5min":3.2189999999999999,"15min":2.6539999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":19.434000000000001,"5min":21.498999999999999,"15min":21.498999999999999},"last":0.52700000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.8899999999999999,"15min":2.1320000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.702,"5min":9.5069999999999997,"15min":21.326000000000001},"last":5.5839999999999996},{"interface":"front","average":{"1min":2.855,"5min":3.0070000000000001,"15min":2.6120000000000001},"min":{"1min":0.30299999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":15.351000000000001,"5min":15.351000000000001,"15min":15.351000000000001},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2985959424,"data_stored":28906027979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4843646976,"data_stored":46918628625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38867,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1859121152,"data_stored":18008418630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:20:43.779 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:43.779 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:43.779 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:43.780 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:20:43.780 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:20:44.476 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.474+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:44.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.567+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:44.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.567+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:44.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.567+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:44.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.573+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:44.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.593+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:44.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.593+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:44.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.593+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:44.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.596+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:44.598 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:44.596+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:48.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:48.841+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:48.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:48.845+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:48.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:48.845+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:48.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:48.845+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:48.979 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:48.977+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:49.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:49.037+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:50.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.508+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:50.585 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:50.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.631+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:50.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.631+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:50.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.689+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:50.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.689+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:50.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.689+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:50.771 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.761+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:50.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:50.830+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:51.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.041+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:51.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.051+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:51.082 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.081+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:51.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.082+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:51.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.082+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:51.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.593+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:51.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.685+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:51.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.685+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:51.733 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.732+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:51.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.735+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:51.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.735+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:51.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.738+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:51.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.747+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:51.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.755+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:51.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.768+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:51.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.801+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:51.965 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.961+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:51.983 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:51.982+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:52.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.062+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:52.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.062+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:52.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.062+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:52.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.063+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:52.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.063+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:52.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.063+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:52.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.063+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:52.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.296+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:52.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.296+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:20:52.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.297+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:52.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.297+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:52.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.299+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:52.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.299+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:52.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.299+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:52.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.348+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:52.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.348+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:52.359 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.358+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:52.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.365+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:52.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:52.384+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:54.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:54.312+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:20:54.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:54.312+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:20:54.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:54.315+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:54.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:54.317+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:54.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:54.372+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:20:57.920 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:20:57.919+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:20:57.928 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:20:57.928+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:20:59.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.261+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:59.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.261+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:59.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.261+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:20:59.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.269+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:20:59.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.331+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:20:59.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.343+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:20:59.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.935+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:20:59.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.935+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:20:59.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.935+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:20:59.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.935+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:00.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:20:59.994+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:00.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.029+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:00.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.029+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:00.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.046+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:00.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.065+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:00.219 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.219+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:00.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.231+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:00.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.342+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:00.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.354+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:00.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.354+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:00.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.354+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:00.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.354+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.413+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.414+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.414+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.414+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:00.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.414+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:00.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.578+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:00.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.610+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:00.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.686+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:00.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.708+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:00.709 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.708+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:00.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.708+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:00.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.728+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:00.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.728+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:00.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.728+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:00.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.771+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:00.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:00.775+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:02.651 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.644+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:02.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.684+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:02.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.697+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:02.701 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.697+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:02.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.715+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:02.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.847+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:02.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.869+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:02.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.869+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:02.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.904+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:02.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.941+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:02.948 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.948+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:02.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:02.948+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:03.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:21:04.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:04.490+0000 7fc515f98700 -1 reset not still connected to 0x5622110dec30 2023-10-03T18:21:04.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:04.490+0000 7fc515f98700 -1 reset not still connected to 0x56221791b860 2023-10-03T18:21:04.569 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:21:04.601 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:21:04.610 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1717,"stamp":"2023-10-03T18:21:04.256480+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23365801777,"num_objects":56847,"num_object_clones":51581,"num_object_copies":113694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56847,"num_whiteouts":0,"num_read":1298432,"num_read_kb":4744282,"num_write":1194055,"num_write_kb":4764363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80569,"ondisk_log_size":80569,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14701876,"kb_used_data":9527512,"kb_used_omap":6100,"kb_used_meta":5168235,"kb_avail":268413644,"statfs":{"total":289910292480,"available":274855571456,"internally_reserved":0,"allocated":9756172288,"data_stored":93398553241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6246424,"internal_metadata":5292273640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,2,10,18,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13795328,"num_objects":-234,"num_object_clones":-234,"num_object_copies":-468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-234,"num_whiteouts":0,"num_read":4587,"num_read_kb":13654,"num_write":3289,"num_write_kb":13156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.029996"},"pg_stats":[{"pgid":"2.7","version":"275'174355","reported_seq":313873,"reported_epoch":275,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:02.643648+0000","last_change":"2023-10-03T18:21:02.474993+0000","last_active":"2023-10-03T18:21:02.643648+0000","last_peered":"2023-10-03T18:21:02.643648+0000","last_clean":"2023-10-03T18:21:02.643648+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:21:02.643648+0000","last_undegraded":"2023-10-03T18:21:02.643648+0000","last_fullsized":"2023-10-03T18:21:02.643648+0000","mapping_epoch":12,"log_start":"248'164291","ondisk_log_start":"248'164291","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":215,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:11:18.883531+0000","scrub_duration":505.38444223300002,"objects_trimmed":2,"snaptrim_duration":2.8916104699999998,"stat_sum":{"num_bytes":2911121842,"num_objects":9711,"num_object_clones":9061,"num_object_copies":19422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9711,"num_whiteouts":0,"num_read":158635,"num_read_kb":590197,"num_write":148397,"num_write_kb":592191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"5"}]},{"pgid":"2.6","version":"275'161479","reported_seq":291020,"reported_epoch":275,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:02.578291+0000","last_change":"2023-10-03T18:21:02.271679+0000","last_active":"2023-10-03T18:21:02.578291+0000","last_peered":"2023-10-03T18:21:02.578291+0000","last_clean":"2023-10-03T18:21:02.578291+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:21:02.578291+0000","last_undegraded":"2023-10-03T18:21:02.578291+0000","last_fullsized":"2023-10-03T18:21:02.578291+0000","mapping_epoch":12,"log_start":"241'151447","ondisk_log_start":"241'151447","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":132,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.9598824839999995,"stat_sum":{"num_bytes":2689598011,"num_objects":6642,"num_object_clones":6039,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6642,"num_whiteouts":0,"num_read":147707,"num_read_kb":545694,"num_write":137123,"num_write_kb":547499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"8"}]},{"pgid":"2.5","version":"275'193961","reported_seq":346414,"reported_epoch":275,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:02.606031+0000","last_change":"2023-10-03T18:21:00.536041+0000","last_active":"2023-10-03T18:21:02.606031+0000","last_peered":"2023-10-03T18:21:02.606031+0000","last_clean":"2023-10-03T18:21:02.606031+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:21:02.606031+0000","last_undegraded":"2023-10-03T18:21:02.606031+0000","last_fullsized":"2023-10-03T18:21:02.606031+0000","mapping_epoch":12,"log_start":"248'183942","ondisk_log_start":"248'183942","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":156,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.2635016110000001,"stat_sum":{"num_bytes":3153932841,"num_objects":8559,"num_object_clones":7851,"num_object_copies":17118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8559,"num_whiteouts":0,"num_read":180058,"num_read_kb":643882,"num_write":161451,"num_write_kb":643388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"4"}]},{"pgid":"2.0","version":"274'182083","reported_seq":327799,"reported_epoch":274,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:01.978485+0000","last_change":"2023-10-03T18:21:00.517708+0000","last_active":"2023-10-03T18:21:01.978485+0000","last_peered":"2023-10-03T18:21:01.978485+0000","last_clean":"2023-10-03T18:21:01.978485+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:21:01.978485+0000","last_undegraded":"2023-10-03T18:21:01.978485+0000","last_fullsized":"2023-10-03T18:21:01.978485+0000","mapping_epoch":12,"log_start":"248'172012","ondisk_log_start":"248'172012","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":231,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:35:05.078733+0000","scrub_duration":734.24538164700004,"objects_trimmed":76,"snaptrim_duration":3.28355123,"stat_sum":{"num_bytes":3047175053,"num_objects":10691,"num_object_clones":10001,"num_object_copies":21382,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10691,"num_whiteouts":0,"num_read":165066,"num_read_kb":615007,"num_write":155846,"num_write_kb":622055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ea","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":436,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-03T18:20:57.363618+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:20:57.363618+0000","last_peered":"2023-10-03T18:20:57.363618+0000","last_clean":"2023-10-03T18:20:57.363618+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:20:57.363618+0000","last_undegraded":"2023-10-03T18:20:57.363618+0000","last_fullsized":"2023-10-03T18:20:57.363618+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"275'186694","reported_seq":333530,"reported_epoch":275,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:02.585301+0000","last_change":"2023-10-03T18:21:02.284893+0000","last_active":"2023-10-03T18:21:02.585301+0000","last_peered":"2023-10-03T18:21:02.585301+0000","last_clean":"2023-10-03T18:21:02.585301+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:21:02.585301+0000","last_undegraded":"2023-10-03T18:21:02.585301+0000","last_fullsized":"2023-10-03T18:21:02.585301+0000","mapping_epoch":12,"log_start":"248'176664","ondisk_log_start":"248'176664","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":149,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":1.0494577350000001,"stat_sum":{"num_bytes":3099267914,"num_objects":8205,"num_object_clones":7506,"num_object_copies":16410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8205,"num_whiteouts":0,"num_read":168256,"num_read_kb":626360,"num_write":158116,"num_write_kb":631063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"1"}]},{"pgid":"2.1","version":"274'180595","reported_seq":314139,"reported_epoch":274,"state":"active+clean","last_fresh":"2023-10-03T18:21:02.009160+0000","last_change":"2023-10-03T18:20:42.279196+0000","last_active":"2023-10-03T18:21:02.009160+0000","last_peered":"2023-10-03T18:21:02.009160+0000","last_clean":"2023-10-03T18:21:02.009160+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:21:02.009160+0000","last_undegraded":"2023-10-03T18:21:02.009160+0000","last_fullsized":"2023-10-03T18:21:02.009160+0000","mapping_epoch":12,"log_start":"248'170524","ondisk_log_start":"248'170524","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:17:50.359035+0000","scrub_duration":424.08306342600002,"objects_trimmed":1,"snaptrim_duration":0.091296477000000001,"stat_sum":{"num_bytes":2823492207,"num_objects":2998,"num_object_clones":2363,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":157613,"num_read_kb":571629,"num_write":144093,"num_write_kb":574573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ea","length":"6"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"}]},{"pgid":"2.2","version":"275'180742","reported_seq":318872,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-03T18:21:03.208292+0000","last_change":"2023-10-03T18:20:46.054974+0000","last_active":"2023-10-03T18:21:03.208292+0000","last_peered":"2023-10-03T18:21:03.208292+0000","last_clean":"2023-10-03T18:21:03.208292+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:21:03.208292+0000","last_undegraded":"2023-10-03T18:21:03.208292+0000","last_fullsized":"2023-10-03T18:21:03.208292+0000","mapping_epoch":12,"log_start":"248'170656","ondisk_log_start":"248'170656","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:05:56.801509+0000","scrub_duration":370.574046178,"objects_trimmed":71,"snaptrim_duration":3.8442226860000002,"stat_sum":{"num_bytes":2827236051,"num_objects":3010,"num_object_clones":2369,"num_object_copies":6020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3010,"num_whiteouts":0,"num_read":165900,"num_read_kb":581824,"num_write":145417,"num_write_kb":579211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"}]},{"pgid":"2.4","version":"275'170645","reported_seq":305942,"reported_epoch":275,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:02.579564+0000","last_change":"2023-10-03T18:21:02.445227+0000","last_active":"2023-10-03T18:21:02.579564+0000","last_peered":"2023-10-03T18:21:02.579564+0000","last_clean":"2023-10-03T18:21:02.579564+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:21:02.579564+0000","last_undegraded":"2023-10-03T18:21:02.579564+0000","last_fullsized":"2023-10-03T18:21:02.579564+0000","mapping_epoch":12,"log_start":"246'160548","ondisk_log_start":"246'160548","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":136,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.16470965000000001,"stat_sum":{"num_bytes":2813387490,"num_objects":7029,"num_object_clones":6391,"num_object_copies":14058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7029,"num_whiteouts":0,"num_read":155091,"num_read_kb":569598,"num_write":143475,"num_write_kb":572489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ed","length":"3"},{"start":"fa","length":"7"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23365211409,"num_objects":56845,"num_object_clones":51581,"num_object_copies":113690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56845,"num_whiteouts":0,"num_read":1298326,"num_read_kb":4744191,"num_write":1193918,"num_write_kb":4762469,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9748738048,"data_stored":93365348898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72144,"internal_metadata":0},"log_size":80470,"ondisk_log_size":80470,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738948,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3751864,"kb_used_data":1828976,"kb_used_omap":969,"kb_used_meta":1921910,"kb_avail":90619976,"statfs":{"total":96636764160,"available":92794855424,"internally_reserved":0,"allocated":1872871424,"data_stored":17917256707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":992402,"internal_metadata":1968036718},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":2.4500000000000002,"5min":2.4039999999999999,"15min":2.5720000000000001},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":7.8440000000000003,"5min":13.859999999999999,"15min":32.173000000000002},"last":1.038},{"interface":"front","average":{"1min":1.3680000000000001,"5min":2.4049999999999998,"15min":2.2210000000000001},"min":{"1min":0.21299999999999999,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":6.7149999999999999,"5min":30.128,"15min":30.128},"last":0.98499999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:20:11 2023","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":2.153,"15min":2.5819999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":4.4610000000000003,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.9950000000000001,"5min":2.0030000000000001,"15min":2.3719999999999999},"min":{"1min":0.247,"5min":0.23000000000000001,"15min":0.188},"max":{"1min":7.5549999999999997,"5min":12.346,"15min":19.097000000000001},"last":0.40200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738947,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3958012,"kb_used_data":2936496,"kb_used_omap":2116,"kb_used_meta":1019387,"kb_avail":90413828,"statfs":{"total":96636764160,"available":92583759872,"internally_reserved":0,"allocated":3006971904,"data_stored":28782554280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1043852799},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.7370000000000001,"15min":3.169},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":12.895,"5min":19.969999999999999,"15min":31.474},"last":2.3690000000000002},{"interface":"front","average":{"1min":2.9900000000000002,"5min":3.0899999999999999,"15min":2.96},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":9.1750000000000007,"5min":19.498999999999999,"15min":30.273},"last":2.1320000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":2.0819999999999999,"15min":1.841},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":5.484,"5min":13.058,"15min":13.209},"last":1.129},{"interface":"front","average":{"1min":1.952,"5min":2.0910000000000002,"15min":2.04},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.1849999999999996,"5min":12.151999999999999,"15min":16.033999999999999},"last":2.1659999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738926,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6992000,"kb_used_data":4762040,"kb_used_omap":3014,"kb_used_meta":2226937,"kb_avail":87379840,"statfs":{"total":96636764160,"available":89476956160,"internally_reserved":0,"allocated":4876328960,"data_stored":46698742254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3086725,"internal_metadata":2280384123},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2,0,9,15,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":4.4219999999999997,"5min":3.4809999999999999,"15min":2.8450000000000002},"min":{"1min":0.25,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":19.361999999999998,"5min":25.145,"15min":25.145},"last":0.30199999999999999},{"interface":"front","average":{"1min":3.9340000000000002,"5min":3.2189999999999999,"15min":2.6539999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":19.434000000000001,"5min":21.498999999999999,"15min":21.498999999999999},"last":0.38}]},{"osd":2,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.8899999999999999,"15min":2.1320000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.702,"5min":9.5069999999999997,"15min":21.326000000000001},"last":11.243},{"interface":"front","average":{"1min":2.855,"5min":3.0070000000000001,"15min":2.6120000000000001},"min":{"1min":0.30299999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":15.351000000000001,"5min":15.351000000000001,"15min":15.351000000000001},"last":0.41599999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3003908096,"data_stored":28778662859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4874231808,"data_stored":46678508817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38867,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1870598144,"data_stored":17908177222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7007,"internal_metadata":0}]}} 2023-10-03T18:21:04.612 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:04.612 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:04.612 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:04.612 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:04.612 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:21:04.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:04.618+0000 7fc515f98700 -1 reset not still connected to 0x562231dc3790 2023-10-03T18:21:04.815 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:04.815+0000 7fc515f98700 -1 reset not still connected to 0x562233181520 2023-10-03T18:21:05.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:05.840+0000 7fc515f98700 -1 reset not still connected to 0x56221b356680 2023-10-03T18:21:05.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:05.871+0000 7fc515f98700 -1 reset not still connected to 0x562231dc3790 2023-10-03T18:21:05.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:05.871+0000 7fc515f98700 -1 reset not still connected to 0x562233181520 2023-10-03T18:21:07.441 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.432+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:07.546 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.542+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:07.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.542+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:07.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.545+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:07.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.545+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:07.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.714+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:07.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.714+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:07.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.721+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:07.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.721+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:07.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:07.769+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:09.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.356+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:09.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.356+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:09.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.502+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:09.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.502+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:09.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.574+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:09.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.633+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:09.660 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:09.658+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:12.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.413+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:12.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.414+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:12.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.431+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:12.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.431+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:12.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.431+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:12.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.459+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:12.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.540+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:12.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.540+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:12.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:12.572+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:14.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:14.979+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:14.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:14.979+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:15.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.064+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:15.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.065+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:15.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.109+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:15.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.109+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:15.110 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.109+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:15.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.157+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:15.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:15.187+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:22.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.187+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:22.205 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.200+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:22.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.240+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:22.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.259+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:22.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.291+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:22.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.291+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:22.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.318+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:22.319 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.318+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:22.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.352+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:22.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.492+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:22.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.620+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:22.623 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.620+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:22.623 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.620+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:22.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.712+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:22.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.712+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:22.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.712+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:22.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.712+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:22.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:22.712+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:24.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:21:25.301 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:21:25.323 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:21:25.343 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1737,"stamp":"2023-10-03T18:21:25.167837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23383865137,"num_objects":58030,"num_object_clones":52764,"num_object_copies":116060,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58030,"num_whiteouts":0,"num_read":1305693,"num_read_kb":4763675,"num_write":1198564,"num_write_kb":4781679,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80394,"ondisk_log_size":80394,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14971480,"kb_used_data":9563572,"kb_used_omap":6501,"kb_used_meta":5401370,"kb_avail":268144040,"statfs":{"total":289910292480,"available":274579496960,"internally_reserved":0,"allocated":9793097728,"data_stored":95986306209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6657521,"internal_metadata":5531003407},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,2,6,6,17,7,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5693440,"num_objects":402,"num_object_clones":402,"num_object_copies":804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":402,"num_whiteouts":0,"num_read":1369,"num_read_kb":5452,"num_write":1362,"num_write_kb":5448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002017"},"pg_stats":[{"pgid":"2.7","version":"285'175315","reported_seq":315079,"reported_epoch":285,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:19.636117+0000","last_change":"2023-10-03T18:21:18.150028+0000","last_active":"2023-10-03T18:21:19.636117+0000","last_peered":"2023-10-03T18:21:19.636117+0000","last_clean":"2023-10-03T18:21:19.636117+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:21:19.636117+0000","last_undegraded":"2023-10-03T18:21:19.636117+0000","last_fullsized":"2023-10-03T18:21:19.636117+0000","mapping_epoch":12,"log_start":"248'165292","ondisk_log_start":"248'165292","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":233,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:33:11.172492+0000","scrub_duration":505.38444223300002,"objects_trimmed":10,"snaptrim_duration":0.47909095400000001,"stat_sum":{"num_bytes":2913190373,"num_objects":9927,"num_object_clones":9276,"num_object_copies":19854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9927,"num_whiteouts":0,"num_read":159388,"num_read_kb":592271,"num_write":148920,"num_write_kb":594215,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"7"}]},{"pgid":"2.6","version":"285'162343","reported_seq":292181,"reported_epoch":285,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:19.612422+0000","last_change":"2023-10-03T18:21:18.143032+0000","last_active":"2023-10-03T18:21:19.612422+0000","last_peered":"2023-10-03T18:21:19.612422+0000","last_clean":"2023-10-03T18:21:19.612422+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:21:19.612422+0000","last_undegraded":"2023-10-03T18:21:19.612422+0000","last_fullsized":"2023-10-03T18:21:19.612422+0000","mapping_epoch":12,"log_start":"248'152252","ondisk_log_start":"248'152252","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":149,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.84057961999999997,"stat_sum":{"num_bytes":2691449403,"num_objects":6810,"num_object_clones":6207,"num_object_copies":13620,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6810,"num_whiteouts":0,"num_read":148366,"num_read_kb":547586,"num_write":137578,"num_write_kb":549283,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"1"}]},{"pgid":"2.5","version":"285'195069","reported_seq":347917,"reported_epoch":285,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:19.635807+0000","last_change":"2023-10-03T18:21:16.297841+0000","last_active":"2023-10-03T18:21:19.635807+0000","last_peered":"2023-10-03T18:21:19.635807+0000","last_clean":"2023-10-03T18:21:19.635807+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:21:19.635807+0000","last_undegraded":"2023-10-03T18:21:19.635807+0000","last_fullsized":"2023-10-03T18:21:19.635807+0000","mapping_epoch":12,"log_start":"248'185050","ondisk_log_start":"248'185050","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":174,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.023157586,"stat_sum":{"num_bytes":3156496937,"num_objects":8812,"num_object_clones":8104,"num_object_copies":17624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8812,"num_whiteouts":0,"num_read":181212,"num_read_kb":646634,"num_write":162087,"num_write_kb":645802,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"6"}]},{"pgid":"2.0","version":"285'183632","reported_seq":329621,"reported_epoch":285,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:22.107016+0000","last_change":"2023-10-03T18:21:20.941698+0000","last_active":"2023-10-03T18:21:22.107016+0000","last_peered":"2023-10-03T18:21:22.107016+0000","last_clean":"2023-10-03T18:21:22.107016+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:21:22.107016+0000","last_undegraded":"2023-10-03T18:21:22.107016+0000","last_fullsized":"2023-10-03T18:21:22.107016+0000","mapping_epoch":12,"log_start":"248'173630","ondisk_log_start":"248'173630","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":242,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:08:30.016466+0000","scrub_duration":734.24538164700004,"objects_trimmed":69,"snaptrim_duration":3.6823924030000001,"stat_sum":{"num_bytes":3049779956,"num_objects":10753,"num_object_clones":10066,"num_object_copies":21506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10753,"num_whiteouts":0,"num_read":165985,"num_read_kb":617717,"num_write":156503,"num_write_kb":624592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":446,"reported_epoch":284,"state":"active+clean","last_fresh":"2023-10-03T18:21:12.267098+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:21:12.267098+0000","last_peered":"2023-10-03T18:21:12.267098+0000","last_clean":"2023-10-03T18:21:12.267098+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:21:12.267098+0000","last_undegraded":"2023-10-03T18:21:12.267098+0000","last_fullsized":"2023-10-03T18:21:12.267098+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"285'187708","reported_seq":334869,"reported_epoch":285,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:19.442409+0000","last_change":"2023-10-03T18:21:19.242694+0000","last_active":"2023-10-03T18:21:19.442409+0000","last_peered":"2023-10-03T18:21:19.442409+0000","last_clean":"2023-10-03T18:21:19.442409+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:21:19.442409+0000","last_undegraded":"2023-10-03T18:21:19.442409+0000","last_fullsized":"2023-10-03T18:21:19.442409+0000","mapping_epoch":12,"log_start":"248'177669","ondisk_log_start":"248'177669","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":166,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":1.9405732009999999,"stat_sum":{"num_bytes":3101496190,"num_objects":8389,"num_object_clones":7689,"num_object_copies":16778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8389,"num_whiteouts":0,"num_read":169083,"num_read_kb":628732,"num_write":158660,"num_write_kb":633157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"4"}]},{"pgid":"2.1","version":"285'182037","reported_seq":315919,"reported_epoch":285,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:22.056012+0000","last_change":"2023-10-03T18:21:20.676366+0000","last_active":"2023-10-03T18:21:22.056012+0000","last_peered":"2023-10-03T18:21:22.056012+0000","last_clean":"2023-10-03T18:21:22.056012+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:21:22.056012+0000","last_undegraded":"2023-10-03T18:21:22.056012+0000","last_fullsized":"2023-10-03T18:21:22.056012+0000","mapping_epoch":12,"log_start":"248'172027","ondisk_log_start":"248'172027","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:12:56.344149+0000","scrub_duration":424.08306342600002,"objects_trimmed":93,"snaptrim_duration":3.4185312429999999,"stat_sum":{"num_bytes":2825937518,"num_objects":3044,"num_object_clones":2409,"num_object_copies":6088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":158703,"num_read_kb":574431,"num_write":144696,"num_write_kb":576877,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"7"},{"start":"2ce","length":"1"}]},{"pgid":"2.2","version":"286'182193","reported_seq":320679,"reported_epoch":286,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:23.212383+0000","last_change":"2023-10-03T18:21:22.096119+0000","last_active":"2023-10-03T18:21:23.212383+0000","last_peered":"2023-10-03T18:21:23.212383+0000","last_clean":"2023-10-03T18:21:23.212383+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:21:23.212383+0000","last_undegraded":"2023-10-03T18:21:23.212383+0000","last_fullsized":"2023-10-03T18:21:23.212383+0000","mapping_epoch":12,"log_start":"248'172164","ondisk_log_start":"248'172164","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:22:40.410987+0000","scrub_duration":370.574046178,"objects_trimmed":20,"snaptrim_duration":4.7989780900000003,"stat_sum":{"num_bytes":2829566877,"num_objects":3070,"num_object_clones":2425,"num_object_copies":6140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3070,"num_whiteouts":0,"num_read":166972,"num_read_kb":584542,"num_write":146013,"num_write_kb":581460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"100","length":"1"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"7"},{"start":"2ce","length":"1"}]},{"pgid":"2.4","version":"285'171533","reported_seq":307140,"reported_epoch":285,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:19.634297+0000","last_change":"2023-10-03T18:21:19.242748+0000","last_active":"2023-10-03T18:21:19.634297+0000","last_peered":"2023-10-03T18:21:19.634297+0000","last_clean":"2023-10-03T18:21:19.634297+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:21:19.634297+0000","last_undegraded":"2023-10-03T18:21:19.634297+0000","last_fullsized":"2023-10-03T18:21:19.634297+0000","mapping_epoch":12,"log_start":"248'161451","ondisk_log_start":"248'161451","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":154,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":3.5233166819999999,"stat_sum":{"num_bytes":2815357515,"num_objects":7223,"num_object_clones":6588,"num_object_copies":14446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7223,"num_whiteouts":0,"num_read":155878,"num_read_kb":571671,"num_write":143970,"num_write_kb":574399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"fe","length":"3"},{"start":"104","length":"2"},{"start":"107","length":"1"},{"start":"10e","length":"a"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23383274769,"num_objects":58028,"num_object_clones":52764,"num_object_copies":116056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58028,"num_whiteouts":0,"num_read":1305587,"num_read_kb":4763584,"num_write":1198427,"num_write_kb":4779785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9784668160,"data_stored":96126982690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":72512,"internal_metadata":0},"log_size":80295,"ondisk_log_size":80295,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738952,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3793676,"kb_used_data":1836996,"kb_used_omap":1370,"kb_used_meta":1955301,"kb_avail":90578164,"statfs":{"total":96636764160,"available":92752039936,"internally_reserved":0,"allocated":1881083904,"data_stored":18307009589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1403499,"internal_metadata":2002228629},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,1,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.601,"15min":2.5739999999999998},"min":{"1min":0.189,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.7519999999999998,"5min":13.859999999999999,"15min":32.173000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.371,"15min":2.2210000000000001},"min":{"1min":0.23899999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":7.1200000000000001,"5min":30.128,"15min":30.128},"last":0.28599999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":3.0800000000000001,"5min":2.3730000000000002,"15min":2.609},"min":{"1min":0.26100000000000001,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.779999999999999,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":2.6619999999999999,"5min":2.2280000000000002,"15min":2.3679999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.188},"max":{"1min":11.821,"5min":12.346,"15min":19.097000000000001},"last":0.995}]}]},{"osd":0,"up_from":8,"seq":34359738951,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4044120,"kb_used_data":2948172,"kb_used_omap":2116,"kb_used_meta":1093819,"kb_avail":90327720,"statfs":{"total":96636764160,"available":92495585280,"internally_reserved":0,"allocated":3018928128,"data_stored":29814855142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1120071167},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":2.3540000000000001,"5min":2.7370000000000001,"15min":3.169},"min":{"1min":0.17399999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":12.895,"5min":19.969999999999999,"15min":31.474},"last":0.35499999999999998},{"interface":"front","average":{"1min":2.9900000000000002,"5min":3.0899999999999999,"15min":2.96},"min":{"1min":0.219,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":9.1750000000000007,"5min":19.498999999999999,"15min":30.273},"last":5.077}]},{"osd":2,"last update":"Tue Oct 3 18:20:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":2.0819999999999999,"15min":1.841},"min":{"1min":0.23799999999999999,"5min":0.192,"15min":0.182},"max":{"1min":5.484,"5min":13.058,"15min":13.209},"last":0.32900000000000001},{"interface":"front","average":{"1min":1.952,"5min":2.0910000000000002,"15min":2.04},"min":{"1min":0.218,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.1849999999999996,"5min":12.151999999999999,"15min":16.033999999999999},"last":5.1239999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738929,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7133684,"kb_used_data":4778404,"kb_used_omap":3014,"kb_used_meta":2352249,"kb_avail":87238156,"statfs":{"total":96636764160,"available":89331871744,"internally_reserved":0,"allocated":4893085696,"data_stored":47864441478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3086725,"internal_metadata":2408703611},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2,6,3,13,6,2],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":4.4219999999999997,"5min":3.4809999999999999,"15min":2.8450000000000002},"min":{"1min":0.25,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":19.361999999999998,"5min":25.145,"15min":25.145},"last":12.99},{"interface":"front","average":{"1min":3.9340000000000002,"5min":3.2189999999999999,"15min":2.6539999999999999},"min":{"1min":0.221,"5min":0.19500000000000001,"15min":0.17799999999999999},"max":{"1min":19.434000000000001,"5min":21.498999999999999,"15min":21.498999999999999},"last":7.6070000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.6639999999999999,"5min":1.8899999999999999,"15min":2.1320000000000001},"min":{"1min":0.252,"5min":0.214,"15min":0.19400000000000001},"max":{"1min":5.702,"5min":9.5069999999999997,"15min":21.326000000000001},"last":1.254},{"interface":"front","average":{"1min":2.855,"5min":3.0070000000000001,"15min":2.6120000000000001},"min":{"1min":0.30299999999999999,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":15.351000000000001,"5min":15.351000000000001,"15min":15.351000000000001},"last":0.96699999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3015790592,"data_stored":29847850033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4890804224,"data_stored":47969543441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38867,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1878073344,"data_stored":18309589216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7375,"internal_metadata":0}]}} 2023-10-03T18:21:25.345 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:25.345 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:25.345 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:25.346 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:25.346 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:21:26.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.703+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:26.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.735+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:26.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.735+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:26.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.735+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:26.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.735+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:26.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.757+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:26.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:26.782+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:27.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.090+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:27.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.244+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:27.270 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.269+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:27.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.317+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:27.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.331+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:27.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.332+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:27.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.332+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:27.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.332+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:27.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.480+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:27.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.512+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:27.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.846+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:27.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.873+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:27.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.988+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:27.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:27.988+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:28.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.030+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:28.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.045+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:28.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.045+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:28.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.065+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:28.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.065+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:28.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.107+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:28.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.988+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:28.993 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:28.993+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:29.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.010+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:29.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.010+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:29.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.011+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:29.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.011+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:29.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.023+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:29.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.180+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:29.264 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.260+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:29.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.341+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:29.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.350+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:29.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.350+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:29.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:29.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:29.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:29.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:29.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:29.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.530+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:29.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.672+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:29.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.700+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:29.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.745+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:29.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.745+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:29.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.745+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:29.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.746+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:29.820 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.816+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:29.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.817+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:21:29.858 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.857+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:29.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.904+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:29.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.921+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:29.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.921+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:29.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.969+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:29.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.969+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:29.974 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.969+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:29.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:29.979+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:30.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:30.102+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:31.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:31.270+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:31.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:31.270+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:31.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:31.378+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:31.430 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:31.428+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:31.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:31.472+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:35.193 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:21:35.191+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:21:35.193 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:21:35.191+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:21:35.202 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:21:35.198+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:21:35.205 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:21:35.204+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:21:35.223 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:21:35.219+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:21:40.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.333+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:40.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.333+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:40.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.416+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:40.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.513+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:40.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.513+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:40.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.513+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:40.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.548+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:40.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.636+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:40.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:40.636+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:45.347 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:21:46.140 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:21:46.145 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:21:46.164 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1751,"stamp":"2023-10-03T18:21:45.178081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23408199473,"num_objects":58485,"num_object_clones":53219,"num_object_copies":116970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58485,"num_whiteouts":0,"num_read":1311292,"num_read_kb":4786019,"num_write":1204379,"num_write_kb":4804939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80541,"ondisk_log_size":80541,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14790256,"kb_used_data":9610900,"kb_used_omap":7124,"kb_used_meta":5172203,"kb_avail":268325264,"statfs":{"total":289910292480,"available":274765070336,"internally_reserved":0,"allocated":9841561600,"data_stored":97087467281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7295605,"internal_metadata":5296336267},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,2,9,10,17,20,11,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8933376,"num_objects":-142,"num_object_clones":-142,"num_object_copies":-284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-142,"num_whiteouts":0,"num_read":2095,"num_read_kb":8360,"num_write":2146,"num_write_kb":8584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003216"},"pg_stats":[{"pgid":"2.7","version":"289'176488","reported_seq":316708,"reported_epoch":289,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:41.790260+0000","last_change":"2023-10-03T18:21:40.416255+0000","last_active":"2023-10-03T18:21:41.790260+0000","last_peered":"2023-10-03T18:21:41.790260+0000","last_clean":"2023-10-03T18:21:41.790260+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:21:41.790260+0000","last_undegraded":"2023-10-03T18:21:41.790260+0000","last_fullsized":"2023-10-03T18:21:41.790260+0000","mapping_epoch":12,"log_start":"248'166397","ondisk_log_start":"248'166397","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":230,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:33:11.172492+0000","scrub_duration":505.38444223300002,"objects_trimmed":24,"snaptrim_duration":4.2561952600000001,"stat_sum":{"num_bytes":2916086245,"num_objects":10037,"num_object_clones":9386,"num_object_copies":20074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10037,"num_whiteouts":0,"num_read":160097,"num_read_kb":595103,"num_write":149619,"num_write_kb":597011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"110","length":"8"}]},{"pgid":"2.6","version":"289'163503","reported_seq":293692,"reported_epoch":289,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:41.736072+0000","last_change":"2023-10-03T18:21:40.930756+0000","last_active":"2023-10-03T18:21:41.736072+0000","last_peered":"2023-10-03T18:21:41.736072+0000","last_clean":"2023-10-03T18:21:41.736072+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:21:41.736072+0000","last_undegraded":"2023-10-03T18:21:41.736072+0000","last_fullsized":"2023-10-03T18:21:41.736072+0000","mapping_epoch":12,"log_start":"248'153453","ondisk_log_start":"248'153453","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":146,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":17,"snaptrim_duration":0.48065186100000001,"stat_sum":{"num_bytes":2694390331,"num_objects":6952,"num_object_clones":6349,"num_object_copies":13904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6952,"num_whiteouts":0,"num_read":148980,"num_read_kb":550042,"num_write":138288,"num_write_kb":552123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"110","length":"8"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"2"}]},{"pgid":"2.5","version":"289'196352","reported_seq":349668,"reported_epoch":289,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:41.697845+0000","last_change":"2023-10-03T18:21:38.615004+0000","last_active":"2023-10-03T18:21:41.697845+0000","last_peered":"2023-10-03T18:21:41.697845+0000","last_clean":"2023-10-03T18:21:41.697845+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:21:41.697845+0000","last_undegraded":"2023-10-03T18:21:41.697845+0000","last_fullsized":"2023-10-03T18:21:41.697845+0000","mapping_epoch":12,"log_start":"248'186253","ondisk_log_start":"248'186253","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":171,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":1.3225368550000001,"stat_sum":{"num_bytes":3159917097,"num_objects":8988,"num_object_clones":8280,"num_object_copies":17976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8988,"num_whiteouts":0,"num_read":181946,"num_read_kb":649562,"num_write":162895,"num_write_kb":649034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"110","length":"8"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"9"}]},{"pgid":"2.0","version":"289'185216","reported_seq":331625,"reported_epoch":289,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:42.167075+0000","last_change":"2023-10-03T18:21:41.339518+0000","last_active":"2023-10-03T18:21:42.167075+0000","last_peered":"2023-10-03T18:21:42.167075+0000","last_clean":"2023-10-03T18:21:42.167075+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:21:42.167075+0000","last_undegraded":"2023-10-03T18:21:42.167075+0000","last_fullsized":"2023-10-03T18:21:42.167075+0000","mapping_epoch":12,"log_start":"255'175139","ondisk_log_start":"255'175139","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":234,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:08:30.016466+0000","scrub_duration":734.24538164700004,"objects_trimmed":79,"snaptrim_duration":4.0614137120000002,"stat_sum":{"num_bytes":3052921588,"num_objects":10707,"num_object_clones":10020,"num_object_copies":21414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10707,"num_whiteouts":0,"num_read":166732,"num_read_kb":620697,"num_write":157263,"num_write_kb":627632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"110","length":"4"}]},{"pgid":"1.0","version":"88'99","reported_seq":450,"reported_epoch":288,"state":"active+clean","last_fresh":"2023-10-03T18:21:32.336491+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:21:32.336491+0000","last_peered":"2023-10-03T18:21:32.336491+0000","last_clean":"2023-10-03T18:21:32.336491+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:21:32.336491+0000","last_undegraded":"2023-10-03T18:21:32.336491+0000","last_fullsized":"2023-10-03T18:21:32.336491+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"289'189019","reported_seq":336619,"reported_epoch":289,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:41.662128+0000","last_change":"2023-10-03T18:21:40.416163+0000","last_active":"2023-10-03T18:21:41.662128+0000","last_peered":"2023-10-03T18:21:41.662128+0000","last_clean":"2023-10-03T18:21:41.662128+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:21:41.662128+0000","last_undegraded":"2023-10-03T18:21:41.662128+0000","last_fullsized":"2023-10-03T18:21:41.662128+0000","mapping_epoch":12,"log_start":"248'178972","ondisk_log_start":"248'178972","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":163,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":34,"snaptrim_duration":3.1353139379999999,"stat_sum":{"num_bytes":3104887678,"num_objects":8572,"num_object_clones":7872,"num_object_copies":17144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8572,"num_whiteouts":0,"num_read":169832,"num_read_kb":631724,"num_write":159454,"num_write_kb":636333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"110","length":"8"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"}]},{"pgid":"2.1","version":"289'183541","reported_seq":317839,"reported_epoch":289,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:42.172385+0000","last_change":"2023-10-03T18:21:40.585843+0000","last_active":"2023-10-03T18:21:42.172385+0000","last_peered":"2023-10-03T18:21:42.172385+0000","last_clean":"2023-10-03T18:21:42.172385+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:21:42.172385+0000","last_undegraded":"2023-10-03T18:21:42.172385+0000","last_fullsized":"2023-10-03T18:21:42.172385+0000","mapping_epoch":12,"log_start":"255'173538","ondisk_log_start":"255'173538","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:12:56.344149+0000","scrub_duration":424.08306342600002,"objects_trimmed":80,"snaptrim_duration":3.3073689989999999,"stat_sum":{"num_bytes":2828608110,"num_objects":2908,"num_object_clones":2273,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":159413,"num_read_kb":577255,"num_write":145331,"num_write_kb":579417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"110","length":"8"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"1"}]},{"pgid":"2.2","version":"290'183713","reported_seq":322595,"reported_epoch":290,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:21:43.396894+0000","last_change":"2023-10-03T18:21:42.836296+0000","last_active":"2023-10-03T18:21:43.396894+0000","last_peered":"2023-10-03T18:21:43.396894+0000","last_clean":"2023-10-03T18:21:43.396894+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:21:43.396894+0000","last_undegraded":"2023-10-03T18:21:43.396894+0000","last_fullsized":"2023-10-03T18:21:43.396894+0000","mapping_epoch":12,"log_start":"255'173674","ondisk_log_start":"255'173674","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:22:40.410987+0000","scrub_duration":370.574046178,"objects_trimmed":15,"snaptrim_duration":0.56735031199999997,"stat_sum":{"num_bytes":2832532381,"num_objects":2975,"num_object_clones":2330,"num_object_copies":5950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":167649,"num_read_kb":587250,"num_write":146728,"num_write_kb":584320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"112","length":"6"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"}]},{"pgid":"2.4","version":"289'172694","reported_seq":308710,"reported_epoch":289,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:21:41.616224+0000","last_change":"2023-10-03T18:21:40.879146+0000","last_active":"2023-10-03T18:21:41.616224+0000","last_peered":"2023-10-03T18:21:41.616224+0000","last_clean":"2023-10-03T18:21:41.616224+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:21:41.616224+0000","last_undegraded":"2023-10-03T18:21:41.616224+0000","last_fullsized":"2023-10-03T18:21:41.616224+0000","mapping_epoch":12,"log_start":"248'162658","ondisk_log_start":"248'162658","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":150,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":34,"snaptrim_duration":3.5936391949999997,"stat_sum":{"num_bytes":2818265675,"num_objects":7344,"num_object_clones":6709,"num_object_copies":14688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7344,"num_whiteouts":0,"num_read":156537,"num_read_kb":574295,"num_write":144664,"num_write_kb":577175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"110","length":"8"},{"start":"121","length":"2"},{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23407609105,"num_objects":58483,"num_object_clones":53219,"num_object_copies":116966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58483,"num_whiteouts":0,"num_read":1311186,"num_read_kb":4785928,"num_write":1204242,"num_write_kb":4803045,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9831723008,"data_stored":97075829282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74664,"internal_metadata":0},"log_size":80442,"ondisk_log_size":80442,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738956,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3599640,"kb_used_data":1845520,"kb_used_omap":1373,"kb_used_meta":1752738,"kb_avail":90772200,"statfs":{"total":96636764160,"available":92950732800,"internally_reserved":0,"allocated":1889812480,"data_stored":18340441365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1406319,"internal_metadata":1794804369},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1,3,1,12,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.601,"15min":2.5739999999999998},"min":{"1min":0.189,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.7519999999999998,"5min":13.859999999999999,"15min":32.173000000000002},"last":3.3149999999999999},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.371,"15min":2.2210000000000001},"min":{"1min":0.23899999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":7.1200000000000001,"5min":30.128,"15min":30.128},"last":1.153}]},{"osd":1,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":3.0800000000000001,"5min":2.3730000000000002,"15min":2.609},"min":{"1min":0.26100000000000001,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.779999999999999,"5min":17.274000000000001,"15min":18.768999999999998},"last":1.196},{"interface":"front","average":{"1min":2.6619999999999999,"5min":2.2280000000000002,"15min":2.3679999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.188},"max":{"1min":11.821,"5min":12.346,"15min":19.097000000000001},"last":4.3129999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738955,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4107960,"kb_used_data":2962092,"kb_used_omap":2116,"kb_used_meta":1143739,"kb_avail":90263880,"statfs":{"total":96636764160,"available":92430213120,"internally_reserved":0,"allocated":3033182208,"data_stored":30201808534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1171189247},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,6,2,5,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":2.5710000000000002,"15min":3.1589999999999998},"min":{"1min":0.17499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":8.6760000000000002,"5min":19.969999999999999,"15min":31.474},"last":0.254},{"interface":"front","average":{"1min":2.8519999999999999,"5min":3.1200000000000001,"15min":3.0459999999999998},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":7.6639999999999997,"5min":19.498999999999999,"15min":30.273},"last":0.55500000000000005}]},{"osd":2,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.7529999999999999,"15min":1.8280000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.182},"max":{"1min":3.02,"5min":13.058,"15min":13.209},"last":0.29499999999999998},{"interface":"front","average":{"1min":2.5790000000000002,"5min":2.3090000000000002,"15min":2.1139999999999999},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":9.5939999999999994,"5min":12.151999999999999,"15min":16.033999999999999},"last":0.62}]}]},{"osd":1,"up_from":8,"seq":34359738933,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7082656,"kb_used_data":4803288,"kb_used_omap":3634,"kb_used_meta":2275725,"kb_avail":87289184,"statfs":{"total":96636764160,"available":89384124416,"internally_reserved":0,"allocated":4918566912,"data_stored":48545217382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3721989,"internal_metadata":2330342651},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2,2,5,11,5,7,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":3.8759999999999999,"5min":3.698,"15min":2.8660000000000001},"min":{"1min":0.255,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":12.99,"5min":25.145,"15min":25.145},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.9910000000000001,"5min":2.8929999999999998,"15min":2.6230000000000002},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.17799999999999999},"max":{"1min":9.516,"5min":21.498999999999999,"15min":21.498999999999999},"last":1.181}]},{"osd":2,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.1059999999999999,"15min":2.0590000000000002},"min":{"1min":0.249,"5min":0.22,"15min":0.19400000000000001},"max":{"1min":11.243,"5min":11.243,"15min":21.326000000000001},"last":3.573},{"interface":"front","average":{"1min":1.7130000000000001,"5min":2.5529999999999999,"15min":2.581},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":13.715,"5min":15.351000000000001,"15min":15.351000000000001},"last":1.091}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3029868544,"data_stored":30200286257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4915097600,"data_stored":48537888017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38199,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1886756864,"data_stored":18337655008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:21:46.166 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:46.166 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:46.166 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:46.167 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:21:46.167 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:21:50.287 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:50.283+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:53.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.930+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:53.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.930+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:53.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.930+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:53.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.931+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:53.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.931+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:53.935 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:53.931+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:54.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.473+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:54.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.482+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:54.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.482+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:54.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.490+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:54.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.490+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:54.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.554+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:54.613 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.612+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:54.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.619+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:54.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.684+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:54.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:54.740+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:55.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.645+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:55.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.654+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:21:55.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.654+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:21:55.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.677+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:55.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.710+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:55.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.710+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:55.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.717+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:55.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:55.792+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:56.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:56.194+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:56.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:56.220+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:56.341 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:56.340+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:57.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:57.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:57.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:57.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:57.397+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:58.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.023+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:21:58.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.130+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:21:58.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.155+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:58.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.192+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:58.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.192+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:58.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.223+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:58.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.245+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:58.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:58.253+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:59.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.498+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:59.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.500+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:59.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.500+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:59.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.500+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:59.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.548+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:21:59.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.548+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:21:59.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.636+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:21:59.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.642+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:21:59.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.642+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:21:59.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.642+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:21:59.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:21:59.739+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:00.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.189+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:00.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.228+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:00.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.302+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:00.303 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.302+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:00.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.308+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:00.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.308+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:00.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.308+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:00.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.358+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:00.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:00.432+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:05.794 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:22:05.793+0000 7f57f88ef700 -1 librbd::ImageWatcher: 0x55724b4d2300 image watch failed: 93949318158336, (107) Transport endpoint is not connected 2023-10-03T18:22:05.794 INFO:tasks.ceph.mgr.x.smithi031.stderr:2023-10-03T18:22:05.793+0000 7f57f88ef700 -1 librbd::Watcher: 0x55724b4d2300 handle_error: handle=93949318158336: (107) Transport endpoint is not connected 2023-10-03T18:22:06.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:22:06.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:22:06.848 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:22:06.868 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1771,"stamp":"2023-10-03T18:22:06.313884+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23437199101,"num_objects":58302,"num_object_clones":53037,"num_object_copies":116604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58302,"num_whiteouts":0,"num_read":1320746,"num_read_kb":4817205,"num_write":1211541,"num_write_kb":4833191,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80621,"ondisk_log_size":80621,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13897224,"kb_used_data":9666980,"kb_used_omap":7126,"kb_used_meta":4223081,"kb_avail":269218296,"statfs":{"total":289910292480,"available":275679535104,"internally_reserved":0,"allocated":9898987520,"data_stored":96998807422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7297797,"internal_metadata":4324435195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,8,1,7,10,17,7,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6774732,"num_objects":-24,"num_object_clones":-23,"num_object_copies":-48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-24,"num_whiteouts":0,"num_read":3879,"num_read_kb":8938,"num_write":1772,"num_write_kb":6692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.041086"},"pg_stats":[{"pgid":"2.7","version":"296'177714","reported_seq":318845,"reported_epoch":296,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:03.193246+0000","last_change":"2023-10-03T18:21:59.498547+0000","last_active":"2023-10-03T18:22:03.193246+0000","last_peered":"2023-10-03T18:22:03.193246+0000","last_clean":"2023-10-03T18:22:03.193246+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:22:03.193246+0000","last_undegraded":"2023-10-03T18:22:03.193246+0000","last_fullsized":"2023-10-03T18:22:03.193246+0000","mapping_epoch":12,"log_start":"253'167700","ondisk_log_start":"253'167700","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":233,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:29:06.705673+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":2.2247047329999998,"stat_sum":{"num_bytes":2919694720,"num_objects":10061,"num_object_clones":9412,"num_object_copies":20122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10061,"num_whiteouts":0,"num_read":161357,"num_read_kb":599233,"num_write":150510,"num_write_kb":600520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"125","length":"1"}]},{"pgid":"2.6","version":"296'164653","reported_seq":295679,"reported_epoch":296,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:03.193473+0000","last_change":"2023-10-03T18:22:02.106190+0000","last_active":"2023-10-03T18:22:03.193473+0000","last_peered":"2023-10-03T18:22:03.193473+0000","last_clean":"2023-10-03T18:22:03.193473+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:22:03.193473+0000","last_undegraded":"2023-10-03T18:22:03.193473+0000","last_fullsized":"2023-10-03T18:22:03.193473+0000","mapping_epoch":12,"log_start":"248'154558","ondisk_log_start":"248'154558","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":149,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":1.669267982,"stat_sum":{"num_bytes":2697745005,"num_objects":6946,"num_object_clones":6342,"num_object_copies":13892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6946,"num_whiteouts":0,"num_read":150044,"num_read_kb":553631,"num_write":139114,"num_write_kb":555388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"125","length":"5"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"}]},{"pgid":"2.5","version":"296'197754","reported_seq":352004,"reported_epoch":296,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:03.148233+0000","last_change":"2023-10-03T18:22:02.106097+0000","last_active":"2023-10-03T18:22:03.148233+0000","last_peered":"2023-10-03T18:22:03.148233+0000","last_clean":"2023-10-03T18:22:03.148233+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:22:03.148233+0000","last_undegraded":"2023-10-03T18:22:03.148233+0000","last_fullsized":"2023-10-03T18:22:03.148233+0000","mapping_epoch":12,"log_start":"255'187655","ondisk_log_start":"255'187655","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":174,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.75019617900000002,"stat_sum":{"num_bytes":3163771533,"num_objects":8962,"num_object_clones":8252,"num_object_copies":17924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8962,"num_whiteouts":0,"num_read":183282,"num_read_kb":653848,"num_write":163847,"num_write_kb":652793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"125","length":"5"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"3"}]},{"pgid":"2.0","version":"295'186921","reported_seq":334084,"reported_epoch":295,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:02.252508+0000","last_change":"2023-10-03T18:22:01.096775+0000","last_active":"2023-10-03T18:22:02.252508+0000","last_peered":"2023-10-03T18:22:02.252508+0000","last_clean":"2023-10-03T18:22:02.252508+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:22:02.252508+0000","last_undegraded":"2023-10-03T18:22:02.252508+0000","last_fullsized":"2023-10-03T18:22:02.252508+0000","mapping_epoch":12,"log_start":"255'176844","ondisk_log_start":"255'176844","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":227,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:13:41.275792+0000","scrub_duration":734.24538164700004,"objects_trimmed":12,"snaptrim_duration":0.812872547,"stat_sum":{"num_bytes":3056714384,"num_objects":10457,"num_object_clones":9772,"num_object_copies":20914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10457,"num_whiteouts":0,"num_read":167785,"num_read_kb":624446,"num_write":158201,"num_write_kb":631347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"124","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":458,"reported_epoch":295,"state":"active+clean","last_fresh":"2023-10-03T18:22:01.527795+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:22:01.527795+0000","last_peered":"2023-10-03T18:22:01.527795+0000","last_clean":"2023-10-03T18:22:01.527795+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:22:01.527795+0000","last_undegraded":"2023-10-03T18:22:01.527795+0000","last_fullsized":"2023-10-03T18:22:01.527795+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"296'190391","reported_seq":338917,"reported_epoch":296,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:02.968599+0000","last_change":"2023-10-03T18:22:01.441927+0000","last_active":"2023-10-03T18:22:02.968599+0000","last_peered":"2023-10-03T18:22:02.968599+0000","last_clean":"2023-10-03T18:22:02.968599+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:22:02.968599+0000","last_undegraded":"2023-10-03T18:22:02.968599+0000","last_fullsized":"2023-10-03T18:22:02.968599+0000","mapping_epoch":12,"log_start":"255'180373","ondisk_log_start":"255'180373","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":166,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.088875416999999998,"stat_sum":{"num_bytes":3108762644,"num_objects":8536,"num_object_clones":7833,"num_object_copies":17072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8536,"num_whiteouts":0,"num_read":171115,"num_read_kb":635972,"num_write":160409,"num_write_kb":640097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"125","length":"5"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"1"}]},{"pgid":"2.1","version":"295'184518","reported_seq":319626,"reported_epoch":295,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:02.246437+0000","last_change":"2023-10-03T18:22:01.295080+0000","last_active":"2023-10-03T18:22:02.246437+0000","last_peered":"2023-10-03T18:22:02.246437+0000","last_clean":"2023-10-03T18:22:02.246437+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:22:02.246437+0000","last_undegraded":"2023-10-03T18:22:02.246437+0000","last_fullsized":"2023-10-03T18:22:02.246437+0000","mapping_epoch":12,"log_start":"255'174438","ondisk_log_start":"255'174438","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:41:09.155894+0000","scrub_duration":424.08306342600002,"objects_trimmed":18,"snaptrim_duration":4.95690542,"stat_sum":{"num_bytes":2832044604,"num_objects":2979,"num_object_clones":2345,"num_object_copies":5958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2979,"num_whiteouts":0,"num_read":160406,"num_read_kb":580763,"num_write":146170,"num_write_kb":582740,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"124","length":"6"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"}]},{"pgid":"2.2","version":"297'184871","reported_seq":324659,"reported_epoch":297,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:03.449070+0000","last_change":"2023-10-03T18:22:02.662892+0000","last_active":"2023-10-03T18:22:03.449070+0000","last_peered":"2023-10-03T18:22:03.449070+0000","last_clean":"2023-10-03T18:22:03.449070+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:22:03.449070+0000","last_undegraded":"2023-10-03T18:22:03.449070+0000","last_fullsized":"2023-10-03T18:22:03.449070+0000","mapping_epoch":12,"log_start":"255'174775","ondisk_log_start":"255'174775","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:17:34.393142+0000","scrub_duration":370.574046178,"objects_trimmed":4,"snaptrim_duration":0.34327316200000002,"stat_sum":{"num_bytes":2836034259,"num_objects":2986,"num_object_clones":2345,"num_object_copies":5972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":169031,"num_read_kb":591115,"num_write":147608,"num_write_kb":587737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"125","length":"5"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2f0","length":"1"}]},{"pgid":"2.4","version":"296'173902","reported_seq":310761,"reported_epoch":296,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:02.962025+0000","last_change":"2023-10-03T18:22:01.463649+0000","last_active":"2023-10-03T18:22:02.962025+0000","last_peered":"2023-10-03T18:22:02.962025+0000","last_clean":"2023-10-03T18:22:02.962025+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:22:02.962025+0000","last_undegraded":"2023-10-03T18:22:02.962025+0000","last_fullsized":"2023-10-03T18:22:02.962025+0000","mapping_epoch":12,"log_start":"253'163859","ondisk_log_start":"253'163859","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10043,"log_dups_size":0,"ondisk_log_size":10043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":154,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.75672153600000003,"stat_sum":{"num_bytes":2821841584,"num_objects":7373,"num_object_clones":6736,"num_object_copies":14746,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7373,"num_whiteouts":0,"num_read":157620,"num_read_kb":578106,"num_write":145545,"num_write_kb":580675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"125","length":"5"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23436608733,"num_objects":58300,"num_object_clones":53037,"num_object_copies":116600,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58300,"num_whiteouts":0,"num_read":1320640,"num_read_kb":4817114,"num_write":1211404,"num_write_kb":4831297,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9888890880,"data_stored":97037433326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80522,"ondisk_log_size":80522,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738960,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3620092,"kb_used_data":1856240,"kb_used_omap":1373,"kb_used_meta":1762466,"kb_avail":90751748,"statfs":{"total":96636764160,"available":92929789952,"internally_reserved":0,"allocated":1900789760,"data_stored":18209777633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1406319,"internal_metadata":1804765841},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,2,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":2.1509999999999998,"5min":2.601,"15min":2.5739999999999998},"min":{"1min":0.189,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.7519999999999998,"5min":13.859999999999999,"15min":32.173000000000002},"last":1.8140000000000001},{"interface":"front","average":{"1min":1.7250000000000001,"5min":2.371,"15min":2.2210000000000001},"min":{"1min":0.23899999999999999,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":7.1200000000000001,"5min":30.128,"15min":30.128},"last":3.3260000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:21:12 2023","interfaces":[{"interface":"back","average":{"1min":3.0800000000000001,"5min":2.3730000000000002,"15min":2.609},"min":{"1min":0.26100000000000001,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.779999999999999,"5min":17.274000000000001,"15min":18.768999999999998},"last":3.3690000000000002},{"interface":"front","average":{"1min":2.6619999999999999,"5min":2.2280000000000002,"15min":2.3679999999999999},"min":{"1min":0.214,"5min":0.214,"15min":0.188},"max":{"1min":11.821,"5min":12.346,"15min":19.097000000000001},"last":5.3209999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738959,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4228760,"kb_used_data":2978768,"kb_used_omap":2116,"kb_used_meta":1247867,"kb_avail":90143080,"statfs":{"total":96636764160,"available":92306513920,"internally_reserved":0,"allocated":3050258432,"data_stored":30293009144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1277816319},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,1,6,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":2.5710000000000002,"15min":3.1589999999999998},"min":{"1min":0.17499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":8.6760000000000002,"5min":19.969999999999999,"15min":31.474},"last":2.1970000000000001},{"interface":"front","average":{"1min":2.8519999999999999,"5min":3.1200000000000001,"15min":3.0459999999999998},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":7.6639999999999997,"5min":19.498999999999999,"15min":30.273},"last":1.488}]},{"osd":2,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.7529999999999999,"15min":1.8280000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.182},"max":{"1min":3.02,"5min":13.058,"15min":13.209},"last":1.954},{"interface":"front","average":{"1min":2.5790000000000002,"5min":2.3090000000000002,"15min":2.1139999999999999},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":9.5939999999999994,"5min":12.151999999999999,"15min":16.033999999999999},"last":1.2809999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738937,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6048372,"kb_used_data":4831972,"kb_used_omap":3636,"kb_used_meta":1212747,"kb_avail":88323468,"statfs":{"total":96636764160,"available":90443231232,"internally_reserved":0,"allocated":4947939328,"data_stored":48496020645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":1241853035},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,7,0,5,7,8,3,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":3.8759999999999999,"5min":3.698,"15min":2.8660000000000001},"min":{"1min":0.255,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":12.99,"5min":25.145,"15min":25.145},"last":14.242000000000001},{"interface":"front","average":{"1min":1.9910000000000001,"5min":2.8929999999999998,"15min":2.6230000000000002},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.17799999999999999},"max":{"1min":9.516,"5min":21.498999999999999,"15min":21.498999999999999},"last":15.766999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.1059999999999999,"15min":2.0590000000000002},"min":{"1min":0.249,"5min":0.22,"15min":0.19400000000000001},"max":{"1min":11.243,"5min":11.243,"15min":21.326000000000001},"last":4.0469999999999997},{"interface":"front","average":{"1min":1.7130000000000001,"5min":2.5529999999999999,"15min":2.581},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":13.715,"5min":15.351000000000001,"15min":15.351000000000001},"last":15.206}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3046723584,"data_stored":30310136829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4944510976,"data_stored":48516732177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1897656320,"data_stored":18210564320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:22:06.869 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:06.870 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:06.870 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:06.870 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:06.870 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:22:12.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.223+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:12.297 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.292+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:12.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.326+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:12.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.384+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:12.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.384+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:12.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.384+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:12.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.384+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:12.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.408+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:12.408 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.408+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:12.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.780+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:12.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.780+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:12.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.813+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:12.834 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.834+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:12.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.879+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:12.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.888+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:12.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.888+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:12.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.954+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:12.987 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:12.987+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:13.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:13.029+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:14.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:14.728+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:14.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:14.728+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:14.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:14.728+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:14.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:14.841+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:14.892 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:14.892+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:15.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.000+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:15.057 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.055+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:15.057 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.055+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:15.085 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.084+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:15.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.090+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:15.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.092+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:15.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.224+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:15.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.224+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:15.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.303+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:15.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.311+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:15.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.372+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:15.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.687+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:15.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.687+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:15.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.708+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:15.773 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.772+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:15.774 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.773+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:15.793 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:15.792+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:17.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.280+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:17.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.291+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:17.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.359+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:17.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.359+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:17.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.451+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:17.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.451+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:17.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:17.455+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:18.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:18.431+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:18.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:18.431+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:22.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:22.896+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:23.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.035+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:23.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.076+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:23.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.147+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:23.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.147+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:23.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.147+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:23.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.147+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:23.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.147+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:23.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.150+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:23.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.291+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:23.332 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.332+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:23.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.430+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:23.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.430+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:23.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.430+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:23.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.438+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:23.442 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:23.441+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:24.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.177+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:24.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.177+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:24.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.177+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:24.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.256+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:24.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.293+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:24.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.295+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:24.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.345+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:24.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.418+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:24.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:24.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:24.426+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:25.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.632+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:25.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.726+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:25.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.782+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:25.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.782+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:25.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.799+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:25.868 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.867+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:25.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.917+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:25.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.917+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:25.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.939+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:25.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:25.972+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:26.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.011+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:26.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.185+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:26.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.187+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:26.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.191+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:26.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.193+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.193+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.193+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.193+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.193+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:26.871 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:22:26.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:26.989+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:27.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.104+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:27.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.262+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:27.287 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.277+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:27.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.277+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:27.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.317+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:27.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.346+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:27.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:27.368+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:27.635 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:22:27.652 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:22:27.666 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1786,"stamp":"2023-10-03T18:22:27.199408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23457691441,"num_objects":60021,"num_object_clones":54755,"num_object_copies":120042,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60021,"num_whiteouts":0,"num_read":1328571,"num_read_kb":4840742,"num_write":1216636,"num_write_kb":4853247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80522,"ondisk_log_size":80522,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14272236,"kb_used_data":9705996,"kb_used_omap":7126,"kb_used_meta":4559081,"kb_avail":268843284,"statfs":{"total":289910292480,"available":275295522816,"internally_reserved":0,"allocated":9938939904,"data_stored":100478198841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7297797,"internal_metadata":4668499195},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,5,1,2,6,7,25,21,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10825728,"num_objects":612,"num_object_clones":612,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":0,"num_read":2914,"num_read_kb":11620,"num_write":2646,"num_write_kb":10584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004998"},"pg_stats":[{"pgid":"2.7","version":"304'178903","reported_seq":320390,"reported_epoch":304,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:25.547952+0000","last_change":"2023-10-03T18:22:22.749389+0000","last_active":"2023-10-03T18:22:25.547952+0000","last_peered":"2023-10-03T18:22:25.547952+0000","last_clean":"2023-10-03T18:22:25.547952+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:22:25.547952+0000","last_undegraded":"2023-10-03T18:22:25.547952+0000","last_fullsized":"2023-10-03T18:22:25.547952+0000","mapping_epoch":12,"log_start":"255'168808","ondisk_log_start":"255'168808","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":243,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:39:06.082418+0000","scrub_duration":505.38444223300002,"objects_trimmed":6,"snaptrim_duration":0.41649015700000003,"stat_sum":{"num_bytes":2922295731,"num_objects":10353,"num_object_clones":9703,"num_object_copies":20706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10353,"num_whiteouts":0,"num_read":162195,"num_read_kb":602131,"num_write":151149,"num_write_kb":603062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"128","length":"1"}]},{"pgid":"2.6","version":"304'165795","reported_seq":297184,"reported_epoch":304,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:25.555131+0000","last_change":"2023-10-03T18:22:20.996126+0000","last_active":"2023-10-03T18:22:25.555131+0000","last_peered":"2023-10-03T18:22:25.555131+0000","last_clean":"2023-10-03T18:22:25.555131+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:22:25.555131+0000","last_undegraded":"2023-10-03T18:22:25.555131+0000","last_fullsized":"2023-10-03T18:22:25.555131+0000","mapping_epoch":12,"log_start":"255'155765","ondisk_log_start":"255'155765","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10030,"log_dups_size":0,"ondisk_log_size":10030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":159,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":8.6381165319999997,"stat_sum":{"num_bytes":2700190317,"num_objects":7222,"num_object_clones":6618,"num_object_copies":14444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7222,"num_whiteouts":0,"num_read":150876,"num_read_kb":556443,"num_write":139709,"num_write_kb":557768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"1"}]},{"pgid":"2.5","version":"304'199122","reported_seq":353861,"reported_epoch":304,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:25.588213+0000","last_change":"2023-10-03T18:22:22.749443+0000","last_active":"2023-10-03T18:22:25.588213+0000","last_peered":"2023-10-03T18:22:25.588213+0000","last_clean":"2023-10-03T18:22:25.588213+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:22:25.588213+0000","last_undegraded":"2023-10-03T18:22:25.588213+0000","last_fullsized":"2023-10-03T18:22:25.588213+0000","mapping_epoch":12,"log_start":"255'189064","ondisk_log_start":"255'189064","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":185,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":32,"snaptrim_duration":6.6620244990000002,"stat_sum":{"num_bytes":3166765709,"num_objects":9302,"num_object_clones":8592,"num_object_copies":18604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9302,"num_whiteouts":0,"num_read":184590,"num_read_kb":657399,"num_write":164602,"num_write_kb":655726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"}]},{"pgid":"2.0","version":"304'188439","reported_seq":335832,"reported_epoch":304,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:22.306101+0000","last_change":"2023-10-03T18:22:21.665867+0000","last_active":"2023-10-03T18:22:22.306101+0000","last_peered":"2023-10-03T18:22:22.306101+0000","last_clean":"2023-10-03T18:22:22.306101+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:22:22.306101+0000","last_undegraded":"2023-10-03T18:22:22.306101+0000","last_fullsized":"2023-10-03T18:22:22.306101+0000","mapping_epoch":12,"log_start":"265'178355","ondisk_log_start":"265'178355","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":234,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:01:19.367131+0000","scrub_duration":734.24538164700004,"objects_trimmed":0,"snaptrim_duration":9.3768791839999999,"stat_sum":{"num_bytes":3059094211,"num_objects":10531,"num_object_clones":9845,"num_object_copies":21062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10531,"num_whiteouts":0,"num_read":168668,"num_read_kb":627098,"num_write":158794,"num_write_kb":633680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":466,"reported_epoch":303,"state":"active+clean","last_fresh":"2023-10-03T18:22:12.396127+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:22:12.396127+0000","last_peered":"2023-10-03T18:22:12.396127+0000","last_clean":"2023-10-03T18:22:12.396127+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:22:12.396127+0000","last_undegraded":"2023-10-03T18:22:12.396127+0000","last_fullsized":"2023-10-03T18:22:12.396127+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"304'191704","reported_seq":340617,"reported_epoch":304,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:25.440133+0000","last_change":"2023-10-03T18:22:24.297843+0000","last_active":"2023-10-03T18:22:25.440133+0000","last_peered":"2023-10-03T18:22:25.440133+0000","last_clean":"2023-10-03T18:22:25.440133+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:22:25.440133+0000","last_undegraded":"2023-10-03T18:22:25.440133+0000","last_fullsized":"2023-10-03T18:22:25.440133+0000","mapping_epoch":12,"log_start":"255'181679","ondisk_log_start":"255'181679","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":176,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":1.9844068400000001,"stat_sum":{"num_bytes":3111621448,"num_objects":8855,"num_object_clones":8156,"num_object_copies":17710,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8855,"num_whiteouts":0,"num_read":172030,"num_read_kb":639166,"num_write":161115,"num_write_kb":642894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"4"}]},{"pgid":"2.1","version":"304'186017","reported_seq":321435,"reported_epoch":304,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:22.352207+0000","last_change":"2023-10-03T18:22:21.107226+0000","last_active":"2023-10-03T18:22:22.352207+0000","last_peered":"2023-10-03T18:22:22.352207+0000","last_clean":"2023-10-03T18:22:22.352207+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:22:22.352207+0000","last_undegraded":"2023-10-03T18:22:22.352207+0000","last_fullsized":"2023-10-03T18:22:22.352207+0000","mapping_epoch":12,"log_start":"255'175956","ondisk_log_start":"255'175956","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:29:40.969345+0000","scrub_duration":424.08306342600002,"objects_trimmed":0,"snaptrim_duration":8.8167120210000007,"stat_sum":{"num_bytes":2834256496,"num_objects":3026,"num_object_clones":2391,"num_object_copies":6052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3026,"num_whiteouts":0,"num_read":161516,"num_read_kb":583653,"num_write":146733,"num_write_kb":584917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"}]},{"pgid":"2.2","version":"304'186320","reported_seq":326364,"reported_epoch":304,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:23.559021+0000","last_change":"2023-10-03T18:22:22.793075+0000","last_active":"2023-10-03T18:22:23.559021+0000","last_peered":"2023-10-03T18:22:23.559021+0000","last_clean":"2023-10-03T18:22:23.559021+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:22:23.559021+0000","last_undegraded":"2023-10-03T18:22:23.559021+0000","last_fullsized":"2023-10-03T18:22:23.559021+0000","mapping_epoch":12,"log_start":"255'176287","ondisk_log_start":"255'176287","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:42:18.098240+0000","scrub_duration":370.574046178,"objects_trimmed":15,"snaptrim_duration":0.49586519899999998,"stat_sum":{"num_bytes":2838254291,"num_objects":3056,"num_object_clones":2415,"num_object_copies":6112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":169982,"num_read_kb":593707,"num_write":148160,"num_write_kb":589909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"}]},{"pgid":"2.4","version":"304'175201","reported_seq":312422,"reported_epoch":304,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:25.498942+0000","last_change":"2023-10-03T18:22:24.298715+0000","last_active":"2023-10-03T18:22:25.498942+0000","last_peered":"2023-10-03T18:22:25.498942+0000","last_clean":"2023-10-03T18:22:25.498942+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:22:25.498942+0000","last_undegraded":"2023-10-03T18:22:25.498942+0000","last_fullsized":"2023-10-03T18:22:25.498942+0000","mapping_epoch":12,"log_start":"255'165164","ondisk_log_start":"255'165164","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":164,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":8.4815624760000006,"stat_sum":{"num_bytes":2824622870,"num_objects":7674,"num_object_clones":7035,"num_object_copies":15348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7674,"num_whiteouts":0,"num_read":158608,"num_read_kb":581054,"num_write":146237,"num_write_kb":583397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"128","length":"2"},{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23457101073,"num_objects":60019,"num_object_clones":54755,"num_object_copies":120038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60019,"num_whiteouts":0,"num_read":1328465,"num_read_kb":4840651,"num_write":1216499,"num_write_kb":4851353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9929244672,"data_stored":100533082658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80423,"ondisk_log_size":80423,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738964,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3689632,"kb_used_data":1863320,"kb_used_omap":1373,"kb_used_meta":1824930,"kb_avail":90682208,"statfs":{"total":96636764160,"available":92858580992,"internally_reserved":0,"allocated":1908039680,"data_stored":18698601016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1406319,"internal_metadata":1868728977},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,0,0,0,1,0,8,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":2.5339999999999998,"15min":2.7090000000000001},"min":{"1min":0.23200000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":15.964,"5min":15.964,"15min":32.173000000000002},"last":1.3640000000000001},{"interface":"front","average":{"1min":2.0030000000000001,"5min":2.3740000000000001,"15min":2.23},"min":{"1min":0.252,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":9.1769999999999996,"5min":30.128,"15min":30.128},"last":0.92900000000000005}]},{"osd":1,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.4990000000000001,"15min":2.5430000000000001},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":12.064,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.96899999999999997},{"interface":"front","average":{"1min":3.0379999999999998,"5min":2.5129999999999999,"15min":2.4129999999999998},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.188},"max":{"1min":10.451000000000001,"5min":12.346,"15min":18.751999999999999},"last":0.29599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738963,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4343108,"kb_used_data":2990456,"kb_used_omap":2116,"kb_used_meta":1350523,"kb_avail":90028732,"statfs":{"total":96636764160,"available":92189421568,"internally_reserved":0,"allocated":3062226944,"data_stored":31508880579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1382936063},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,4,1,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":1.796,"5min":2.5710000000000002,"15min":3.1589999999999998},"min":{"1min":0.17499999999999999,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":8.6760000000000002,"5min":19.969999999999999,"15min":31.474},"last":5.9889999999999999},{"interface":"front","average":{"1min":2.8519999999999999,"5min":3.1200000000000001,"15min":3.0459999999999998},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":7.6639999999999997,"5min":19.498999999999999,"15min":30.273},"last":8.7620000000000005}]},{"osd":2,"last update":"Tue Oct 3 18:21:40 2023","interfaces":[{"interface":"back","average":{"1min":0.99199999999999999,"5min":1.7529999999999999,"15min":1.8280000000000001},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.182},"max":{"1min":3.02,"5min":13.058,"15min":13.209},"last":6.1159999999999997},{"interface":"front","average":{"1min":2.5790000000000002,"5min":2.3090000000000002,"15min":2.1139999999999999},"min":{"1min":0.22700000000000001,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":9.5939999999999994,"5min":12.151999999999999,"15min":16.033999999999999},"last":0.28000000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738941,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6239496,"kb_used_data":4852220,"kb_used_omap":3636,"kb_used_meta":1383627,"kb_avail":88132344,"statfs":{"total":96636764160,"available":90247520256,"internally_reserved":0,"allocated":4968673280,"data_stored":50270717246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":1416834155},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,1,0,3,6,13,17,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":3.8759999999999999,"5min":3.698,"15min":2.8660000000000001},"min":{"1min":0.255,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":12.99,"5min":25.145,"15min":25.145},"last":0.19800000000000001},{"interface":"front","average":{"1min":1.9910000000000001,"5min":2.8929999999999998,"15min":2.6230000000000002},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.17799999999999999},"max":{"1min":9.516,"5min":21.498999999999999,"15min":21.498999999999999},"last":0.316}]},{"osd":2,"last update":"Tue Oct 3 18:21:25 2023","interfaces":[{"interface":"back","average":{"1min":2.2050000000000001,"5min":2.1059999999999999,"15min":2.0590000000000002},"min":{"1min":0.249,"5min":0.22,"15min":0.19400000000000001},"max":{"1min":11.243,"5min":11.243,"15min":21.326000000000001},"last":0.23599999999999999},{"interface":"front","average":{"1min":1.7130000000000001,"5min":2.5529999999999999,"15min":2.581},"min":{"1min":0.26300000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":13.715,"5min":15.351000000000001,"15min":15.351000000000001},"last":1.7370000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3058581504,"data_stored":31522221206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4965859328,"data_stored":50307814673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1904803840,"data_stored":18703046779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:22:27.668 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:27.668 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:27.668 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:27.668 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:27.668 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:22:28.324 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:22:28.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:22:29.063 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:22:29.084 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:22:29.710 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:22:29.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:22:30.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.053+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:30.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.053+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:30.188 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.186+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:30.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.220+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:30.469 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:22:30.492 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:22:30.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.624+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:30.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.790+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:30.891 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:30.891+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:30.986 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:22:30.977+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:22:31.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.429+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:31.648 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.644+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:31.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.644+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:31.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.651+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:31.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.666+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:31.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.666+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:31.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.803+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:31.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.803+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:31.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.803+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:31.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:31.827+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:32.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.043+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:32.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.043+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:32.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.043+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:32.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.043+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:32.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.043+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:32.086 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.082+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:32.086 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.082+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:32.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.095+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:32.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.095+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:32.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.108+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:32.279 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.278+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:32.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:32.303+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:33.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.278+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:33.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.278+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:33.286 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.278+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:33.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.304+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:33.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.352+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:33.487 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.485+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:33.487 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.485+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:33.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.486+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:33.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.497+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:33.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:33.714+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:35.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:35.166+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:35.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:35.166+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:35.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:35.297+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:35.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:35.297+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:35.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:35.310+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:38.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.449+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:38.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.481+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:38.490 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.481+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:38.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.518+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.518+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.518+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:38.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.518+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:38.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.600+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:38.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.834+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:38.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.849+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:38.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:38.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.968+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:38.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.968+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:38.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.971+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:38.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.971+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:38.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:38.971+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:39.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:39.756+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:39.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:39.756+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:39.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:39.761+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:39.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:39.798+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:39.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:39.798+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:40.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.019+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:40.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:40.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:40.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:40.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:40.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:40.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:40.058+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:43.414 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.407+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:43.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.453+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:43.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.484+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:43.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.484+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:43.499 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.492+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:43.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.507+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:43.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.619+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:43.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.619+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:43.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.637+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:43.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.637+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:43.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.745+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:43.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.790+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:43.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.822+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:43.893 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.891+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:43.909 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.903+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:43.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.908+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:43.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.908+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:43.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.915+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:43.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.915+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:43.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.925+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:43.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:43.939+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:44.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.025+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:44.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.369+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:44.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.370+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:44.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.413+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:44.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.417+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:44.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.417+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:44.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.463+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:44.472 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.469+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:44.472 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.471+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:44.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.474+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:44.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:44.642+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:46.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:46.926+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:46.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:46.942+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:46.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:46.942+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:46.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:46.942+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:47.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.110+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:47.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.122+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:47.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.152+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:47.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.152+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:47.157 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.152+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:47.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.359+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:47.427 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.419+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:47.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.458+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.458+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.458+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:47.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.458+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:47.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.486+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:47.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.567+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:47.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.577+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:47.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.577+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:47.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.577+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:47.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.584+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:47.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.587+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:47.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.615+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:22:47.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.662+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:22:47.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:47.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.753+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:47.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.796+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:47.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.796+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:47.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.803+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:47.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.803+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:47.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.803+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:47.833 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.823+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:47.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:47.861+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:48.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.459+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:48.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.459+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:48.466 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.461+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:48.479 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.478+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:48.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.536+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:48.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:48.549+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:50.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.456+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:50.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.463+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:50.494 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:22:50.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.566+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:50.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.566+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:50.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.566+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:50.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.566+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:50.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:50.566+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:51.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.072+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:22:51.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.078+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:22:51.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.078+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:51.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.135+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:51.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.135+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:51.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.137+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:51.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.165+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:51.206 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:22:51.244 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:22:51.253 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.250+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:22:51.263 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1799,"stamp":"2023-10-03T18:22:49.209781+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23484802865,"num_objects":59986,"num_object_clones":54720,"num_object_copies":119972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59986,"num_whiteouts":0,"num_read":1335665,"num_read_kb":4868962,"num_write":1223201,"num_write_kb":4879507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80413,"ondisk_log_size":80413,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14227468,"kb_used_data":9758896,"kb_used_omap":7126,"kb_used_meta":4461417,"kb_avail":268888052,"statfs":{"total":289910292480,"available":275341365248,"internally_reserved":0,"allocated":9993109504,"data_stored":100866247049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7297797,"internal_metadata":4568491259},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,1,1,5,6,15,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13832192,"num_objects":-62,"num_object_clones":-62,"num_object_copies":-124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-62,"num_whiteouts":0,"num_read":3671,"num_read_kb":14600,"num_write":3332,"num_write_kb":13328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005627"},"pg_stats":[{"pgid":"2.7","version":"305'180115","reported_seq":322227,"reported_epoch":305,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:46.849585+0000","last_change":"2023-10-03T18:22:44.797686+0000","last_active":"2023-10-03T18:22:46.849585+0000","last_peered":"2023-10-03T18:22:46.849585+0000","last_clean":"2023-10-03T18:22:46.849585+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:22:46.849585+0000","last_undegraded":"2023-10-03T18:22:46.849585+0000","last_fullsized":"2023-10-03T18:22:46.849585+0000","mapping_epoch":12,"log_start":"255'170113","ondisk_log_start":"255'170113","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":240,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:39:06.082418+0000","scrub_duration":505.38444223300002,"objects_trimmed":88,"snaptrim_duration":9.4545980580000002,"stat_sum":{"num_bytes":2925474227,"num_objects":10403,"num_object_clones":9753,"num_object_copies":20806,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10403,"num_whiteouts":0,"num_read":163029,"num_read_kb":605463,"num_write":151925,"num_write_kb":606166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"}]},{"pgid":"2.6","version":"305'166906","reported_seq":298881,"reported_epoch":305,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:46.893368+0000","last_change":"2023-10-03T18:22:46.073518+0000","last_active":"2023-10-03T18:22:46.893368+0000","last_peered":"2023-10-03T18:22:46.893368+0000","last_clean":"2023-10-03T18:22:46.893368+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:22:46.893368+0000","last_undegraded":"2023-10-03T18:22:46.893368+0000","last_fullsized":"2023-10-03T18:22:46.893368+0000","mapping_epoch":12,"log_start":"255'156869","ondisk_log_start":"255'156869","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":155,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":82,"snaptrim_duration":13.744624191,"stat_sum":{"num_bytes":2703155821,"num_objects":7251,"num_object_clones":6647,"num_object_copies":14502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7251,"num_whiteouts":0,"num_read":151642,"num_read_kb":559487,"num_write":140430,"num_write_kb":560652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"5"}]},{"pgid":"2.5","version":"305'200438","reported_seq":355865,"reported_epoch":305,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:46.875637+0000","last_change":"2023-10-03T18:22:44.774961+0000","last_active":"2023-10-03T18:22:46.875637+0000","last_peered":"2023-10-03T18:22:46.875637+0000","last_clean":"2023-10-03T18:22:46.875637+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:22:46.875637+0000","last_undegraded":"2023-10-03T18:22:46.875637+0000","last_fullsized":"2023-10-03T18:22:46.875637+0000","mapping_epoch":12,"log_start":"259'190370","ondisk_log_start":"259'190370","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":181,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":81,"snaptrim_duration":12.414329596,"stat_sum":{"num_bytes":3170341517,"num_objects":9382,"num_object_clones":8672,"num_object_copies":18764,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9382,"num_whiteouts":0,"num_read":185475,"num_read_kb":660927,"num_write":165473,"num_write_kb":659210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"}]},{"pgid":"2.0","version":"306'190377","reported_seq":338465,"reported_epoch":306,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:47.497062+0000","last_change":"2023-10-03T18:22:46.445411+0000","last_active":"2023-10-03T18:22:47.497062+0000","last_peered":"2023-10-03T18:22:47.497062+0000","last_clean":"2023-10-03T18:22:47.497062+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:22:47.497062+0000","last_undegraded":"2023-10-03T18:22:47.497062+0000","last_fullsized":"2023-10-03T18:22:47.497062+0000","mapping_epoch":12,"log_start":"270'180364","ondisk_log_start":"270'180364","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":223,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:01:19.367131+0000","scrub_duration":734.24538164700004,"objects_trimmed":4,"snaptrim_duration":14.123247483,"stat_sum":{"num_bytes":3063063235,"num_objects":10380,"num_object_clones":9694,"num_object_copies":20760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10380,"num_whiteouts":0,"num_read":169714,"num_read_kb":631254,"num_write":159744,"num_write_kb":637480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12c","length":"1"},{"start":"136","length":"9"},{"start":"148","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":467,"reported_epoch":304,"state":"active+clean","last_fresh":"2023-10-03T18:22:22.334363+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:22:22.334363+0000","last_peered":"2023-10-03T18:22:22.334363+0000","last_clean":"2023-10-03T18:22:22.334363+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:22:22.334363+0000","last_undegraded":"2023-10-03T18:22:22.334363+0000","last_fullsized":"2023-10-03T18:22:22.334363+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"305'192935","reported_seq":342517,"reported_epoch":305,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:22:46.854875+0000","last_change":"2023-10-03T18:22:46.107195+0000","last_active":"2023-10-03T18:22:46.854875+0000","last_peered":"2023-10-03T18:22:46.854875+0000","last_clean":"2023-10-03T18:22:46.854875+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:22:46.854875+0000","last_undegraded":"2023-10-03T18:22:46.854875+0000","last_fullsized":"2023-10-03T18:22:46.854875+0000","mapping_epoch":12,"log_start":"255'182883","ondisk_log_start":"255'182883","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":173,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":66,"snaptrim_duration":10.269390377000001,"stat_sum":{"num_bytes":3115041608,"num_objects":8930,"num_object_clones":8231,"num_object_copies":17860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8930,"num_whiteouts":0,"num_read":172908,"num_read_kb":642662,"num_write":161937,"num_write_kb":646182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"7"}]},{"pgid":"2.1","version":"306'187613","reported_seq":323714,"reported_epoch":306,"state":"active+clean","last_fresh":"2023-10-03T18:22:47.540355+0000","last_change":"2023-10-03T18:22:39.584737+0000","last_active":"2023-10-03T18:22:47.540355+0000","last_peered":"2023-10-03T18:22:47.540355+0000","last_clean":"2023-10-03T18:22:47.540355+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:22:47.540355+0000","last_undegraded":"2023-10-03T18:22:47.540355+0000","last_fullsized":"2023-10-03T18:22:47.540355+0000","mapping_epoch":12,"log_start":"265'177559","ondisk_log_start":"265'177559","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:29:40.969345+0000","scrub_duration":424.08306342600002,"objects_trimmed":127,"snaptrim_duration":7.2629318669999998,"stat_sum":{"num_bytes":2837738096,"num_objects":2962,"num_object_clones":2327,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":162470,"num_read_kb":587433,"num_write":147583,"num_write_kb":588317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12c","length":"1"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"}]},{"pgid":"2.2","version":"306'187971","reported_seq":328675,"reported_epoch":306,"state":"active+clean","last_fresh":"2023-10-03T18:22:48.753358+0000","last_change":"2023-10-03T18:22:42.386744+0000","last_active":"2023-10-03T18:22:48.753358+0000","last_peered":"2023-10-03T18:22:48.753358+0000","last_clean":"2023-10-03T18:22:48.753358+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:22:48.753358+0000","last_undegraded":"2023-10-03T18:22:48.753358+0000","last_fullsized":"2023-10-03T18:22:48.753358+0000","mapping_epoch":12,"log_start":"266'177897","ondisk_log_start":"266'177897","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:42:18.098240+0000","scrub_duration":370.574046178,"objects_trimmed":157,"snaptrim_duration":10.073542477,"stat_sum":{"num_bytes":2841903827,"num_objects":2966,"num_object_clones":2325,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":170916,"num_read_kb":597427,"num_write":149035,"num_write_kb":593409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12c","length":"1"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"}]},{"pgid":"2.4","version":"305'176283","reported_seq":314139,"reported_epoch":305,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:22:46.848050+0000","last_change":"2023-10-03T18:22:43.997226+0000","last_active":"2023-10-03T18:22:46.848050+0000","last_peered":"2023-10-03T18:22:46.848050+0000","last_clean":"2023-10-03T18:22:46.848050+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:22:46.848050+0000","last_undegraded":"2023-10-03T18:22:46.848050+0000","last_fullsized":"2023-10-03T18:22:46.848050+0000","mapping_epoch":12,"log_start":"255'166269","ondisk_log_start":"255'166269","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":160,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":85,"snaptrim_duration":11.674308916999999,"stat_sum":{"num_bytes":2827494166,"num_objects":7710,"num_object_clones":7071,"num_object_copies":15420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7710,"num_whiteouts":0,"num_read":159405,"num_read_kb":584218,"num_write":146937,"num_write_kb":586197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"2"},{"start":"136","length":"9"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23484212497,"num_objects":59984,"num_object_clones":54720,"num_object_copies":119968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59984,"num_whiteouts":0,"num_read":1335559,"num_read_kb":4868871,"num_write":1223064,"num_write_kb":4877613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9984143360,"data_stored":100867803682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80314,"ondisk_log_size":80314,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738969,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3780204,"kb_used_data":1874148,"kb_used_omap":1373,"kb_used_meta":1904674,"kb_avail":90591636,"statfs":{"total":96636764160,"available":92765835264,"internally_reserved":0,"allocated":1919127552,"data_stored":18667256520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1406319,"internal_metadata":1950386833},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":2.5339999999999998,"15min":2.7090000000000001},"min":{"1min":0.23200000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":15.964,"5min":15.964,"15min":32.173000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":2.0030000000000001,"5min":2.3740000000000001,"15min":2.23},"min":{"1min":0.252,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":9.1769999999999996,"5min":30.128,"15min":30.128},"last":1.2270000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.4990000000000001,"15min":2.5430000000000001},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":12.064,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.27600000000000002},{"interface":"front","average":{"1min":3.0379999999999998,"5min":2.5129999999999999,"15min":2.4129999999999998},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.188},"max":{"1min":10.451000000000001,"5min":12.346,"15min":18.751999999999999},"last":2.3170000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738968,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4061128,"kb_used_data":3008060,"kb_used_omap":2116,"kb_used_meta":1050939,"kb_avail":90310712,"statfs":{"total":96636764160,"available":92478169088,"internally_reserved":0,"allocated":3080253440,"data_stored":31775159619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1076162047},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":2.4009999999999998,"15min":3.097},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":5.9889999999999999,"5min":19.969999999999999,"15min":31.474},"last":8.8450000000000006},{"interface":"front","average":{"1min":2.883,"5min":3.2229999999999999,"15min":3.024},"min":{"1min":0.25900000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":10.037000000000001,"5min":19.498999999999999,"15min":30.273},"last":4.4710000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.331,"5min":1.806,"15min":1.8899999999999999},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":12.752000000000001,"5min":13.058,"15min":13.209},"last":5.202},{"interface":"front","average":{"1min":2.621,"5min":2.4220000000000002,"15min":2.1469999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":13.459,"5min":13.459,"15min":16.033999999999999},"last":4.3499999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738945,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6386136,"kb_used_data":4876688,"kb_used_omap":3636,"kb_used_meta":1505803,"kb_avail":87985704,"statfs":{"total":96636764160,"available":90097360896,"internally_reserved":0,"allocated":4993728512,"data_stored":50423830910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":1541942379},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,5,12,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:22:25 2023","interfaces":[{"interface":"back","average":{"1min":4.5090000000000003,"5min":4.2169999999999996,"15min":2.98},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":30.896999999999998,"5min":30.896999999999998,"15min":30.896999999999998},"last":2.077},{"interface":"front","average":{"1min":2.5409999999999999,"5min":2.9199999999999999,"15min":2.6219999999999999},"min":{"1min":0.316,"5min":0.19900000000000001,"15min":0.17799999999999999},"max":{"1min":15.766999999999999,"5min":21.498999999999999,"15min":21.498999999999999},"last":2.5310000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:22:25 2023","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":2.3490000000000002,"15min":2.0579999999999998},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":13.358000000000001,"5min":13.358000000000001,"15min":21.326000000000001},"last":8.907},{"interface":"front","average":{"1min":2.5409999999999999,"5min":2.5489999999999999,"15min":2.5880000000000001},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":15.206,"5min":15.351000000000001,"15min":15.351000000000001},"last":2.4940000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3077046272,"data_stored":31775767702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4991107072,"data_stored":50429338897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1915990016,"data_stored":18662697083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:22:51.264 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:51.264 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:51.265 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:51.265 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:22:51.265 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:22:51.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.303+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:51.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.310+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:51.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.310+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:51.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.310+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:51.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.336+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:51.359 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:51.356+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:52.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.353+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:52.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.353+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:52.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.432+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:52.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.523+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:52.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.523+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:52.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.719+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:22:52.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.735+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:22:52.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.735+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:22:52.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.737+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:52.754 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.752+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:52.768 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:52.763+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:22:57.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:56.994+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:22:57.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:57.025+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:22:57.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:22:57.059+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:00.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:00.932+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:00.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:00.936+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:00.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:00.936+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:00.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:00.972+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:00.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:00.975+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:03.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.137+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:23:03.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.205+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:03.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.205+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:03.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.264+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:03.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.330+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:03.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.370+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:03.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.370+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:03.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.378+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:03.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.379+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:03.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:03.386+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:06.255 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:06.254+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:23:06.272 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:06.267+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:23:10.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:10.942+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:11.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.007+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:11.057 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.055+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:11.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.096+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:11.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.096+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:11.102 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.102+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:11.103 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.102+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:11.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:11.124+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:11.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:23:11.649 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:11.648+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:23:11.676 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:11.675+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:23:11.676 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:11.675+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:23:11.996 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:23:12.012 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:23:12.024 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1819,"stamp":"2023-10-03T18:23:11.225127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23513057073,"num_objects":60253,"num_object_clones":54987,"num_object_copies":120506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60253,"num_whiteouts":0,"num_read":1345894,"num_read_kb":4899971,"num_write":1230084,"num_write_kb":4906319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80621,"ondisk_log_size":80621,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14567936,"kb_used_data":9812512,"kb_used_omap":7126,"kb_used_meta":4748265,"kb_avail":268547584,"statfs":{"total":289910292480,"available":274992726016,"internally_reserved":0,"allocated":10048012288,"data_stored":101514613809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7297797,"internal_metadata":4862223611},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,3,6,7,21,10,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5390236,"num_objects":550,"num_object_clones":552,"num_object_copies":1100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":0,"num_read":2838,"num_read_kb":6714,"num_write":1370,"num_write_kb":5059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002754"},"pg_stats":[{"pgid":"2.7","version":"315'181402","reported_seq":324320,"reported_epoch":315,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:08.893075+0000","last_change":"2023-10-03T18:23:03.492145+0000","last_active":"2023-10-03T18:23:08.893075+0000","last_peered":"2023-10-03T18:23:08.893075+0000","last_clean":"2023-10-03T18:23:08.893075+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:23:08.893075+0000","last_undegraded":"2023-10-03T18:23:08.893075+0000","last_fullsized":"2023-10-03T18:23:08.893075+0000","mapping_epoch":12,"log_start":"264'171317","ondisk_log_start":"264'171317","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":256,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:00:10.053676+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.103782745,"stat_sum":{"num_bytes":2929193495,"num_objects":10502,"num_object_clones":9850,"num_object_copies":21004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10502,"num_whiteouts":0,"num_read":164240,"num_read_kb":609382,"num_write":152816,"num_write_kb":609659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"1"}]},{"pgid":"2.6","version":"315'168074","reported_seq":300827,"reported_epoch":315,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:08.948556+0000","last_change":"2023-10-03T18:23:01.560210+0000","last_active":"2023-10-03T18:23:08.948556+0000","last_peered":"2023-10-03T18:23:08.948556+0000","last_clean":"2023-10-03T18:23:08.948556+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:23:08.948556+0000","last_undegraded":"2023-10-03T18:23:08.948556+0000","last_fullsized":"2023-10-03T18:23:08.948556+0000","mapping_epoch":12,"log_start":"255'158072","ondisk_log_start":"255'158072","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":172,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.131127192,"stat_sum":{"num_bytes":2706461244,"num_objects":7320,"num_object_clones":6717,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7320,"num_whiteouts":0,"num_read":152698,"num_read_kb":562996,"num_write":141240,"num_write_kb":563852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"8"}]},{"pgid":"2.5","version":"315'201861","reported_seq":358256,"reported_epoch":315,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:08.948177+0000","last_change":"2023-10-03T18:23:05.595984+0000","last_active":"2023-10-03T18:23:08.948177+0000","last_peered":"2023-10-03T18:23:08.948177+0000","last_clean":"2023-10-03T18:23:08.948177+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:23:08.948177+0000","last_undegraded":"2023-10-03T18:23:08.948177+0000","last_fullsized":"2023-10-03T18:23:08.948177+0000","mapping_epoch":12,"log_start":"268'191779","ondisk_log_start":"268'191779","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":198,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.16546978200000001,"stat_sum":{"num_bytes":3174113936,"num_objects":9427,"num_object_clones":8717,"num_object_copies":18854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9427,"num_whiteouts":0,"num_read":187060,"num_read_kb":665419,"num_write":166434,"num_write_kb":662910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"2"}]},{"pgid":"2.0","version":"315'192027","reported_seq":340866,"reported_epoch":315,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:07.531204+0000","last_change":"2023-10-03T18:23:05.704660+0000","last_active":"2023-10-03T18:23:07.531204+0000","last_peered":"2023-10-03T18:23:07.531204+0000","last_clean":"2023-10-03T18:23:07.531204+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:23:07.531204+0000","last_undegraded":"2023-10-03T18:23:07.531204+0000","last_fullsized":"2023-10-03T18:23:07.531204+0000","mapping_epoch":12,"log_start":"274'181971","ondisk_log_start":"274'181971","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":232,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:04:44.882047+0000","scrub_duration":734.24538164700004,"objects_trimmed":2,"snaptrim_duration":0.31516975000000003,"stat_sum":{"num_bytes":3066495631,"num_objects":10157,"num_object_clones":9472,"num_object_copies":20314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10157,"num_whiteouts":0,"num_read":170921,"num_read_kb":635158,"num_write":160554,"num_write_kb":640651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":477,"reported_epoch":314,"state":"active+clean","last_fresh":"2023-10-03T18:23:06.495923+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:23:06.495923+0000","last_peered":"2023-10-03T18:23:06.495923+0000","last_clean":"2023-10-03T18:23:06.495923+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:23:06.495923+0000","last_undegraded":"2023-10-03T18:23:06.495923+0000","last_fullsized":"2023-10-03T18:23:06.495923+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"315'194326","reported_seq":344730,"reported_epoch":315,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:08.828534+0000","last_change":"2023-10-03T18:23:03.494309+0000","last_active":"2023-10-03T18:23:08.828534+0000","last_peered":"2023-10-03T18:23:08.828534+0000","last_clean":"2023-10-03T18:23:08.828534+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:23:08.828534+0000","last_undegraded":"2023-10-03T18:23:08.828534+0000","last_fullsized":"2023-10-03T18:23:08.828534+0000","mapping_epoch":12,"log_start":"266'184286","ondisk_log_start":"266'184286","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":190,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":2.1887491749999999,"stat_sum":{"num_bytes":3119002442,"num_objects":9008,"num_object_clones":8309,"num_object_copies":18016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9008,"num_whiteouts":0,"num_read":174149,"num_read_kb":646647,"num_write":162887,"num_write_kb":649903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"}]},{"pgid":"2.1","version":"315'188757","reported_seq":325674,"reported_epoch":315,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:07.548880+0000","last_change":"2023-10-03T18:23:05.333642+0000","last_active":"2023-10-03T18:23:07.548880+0000","last_peered":"2023-10-03T18:23:07.548880+0000","last_clean":"2023-10-03T18:23:07.548880+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:23:07.548880+0000","last_undegraded":"2023-10-03T18:23:07.548880+0000","last_fullsized":"2023-10-03T18:23:07.548880+0000","mapping_epoch":12,"log_start":"268'178661","ondisk_log_start":"268'178661","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:17:53.211569+0000","scrub_duration":424.08306342600002,"objects_trimmed":0,"snaptrim_duration":0.92960222100000001,"stat_sum":{"num_bytes":2841215699,"num_objects":3035,"num_object_clones":2398,"num_object_copies":6070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3035,"num_whiteouts":0,"num_read":163751,"num_read_kb":591177,"num_write":148441,"num_write_kb":591635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"2"}]},{"pgid":"2.2","version":"315'189066","reported_seq":330621,"reported_epoch":315,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:08.826094+0000","last_change":"2023-10-03T18:23:08.695890+0000","last_active":"2023-10-03T18:23:08.826094+0000","last_peered":"2023-10-03T18:23:08.826094+0000","last_clean":"2023-10-03T18:23:08.826094+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:23:08.826094+0000","last_undegraded":"2023-10-03T18:23:08.826094+0000","last_fullsized":"2023-10-03T18:23:08.826094+0000","mapping_epoch":12,"log_start":"269'178999","ondisk_log_start":"269'178999","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:42:49.396697+0000","scrub_duration":370.574046178,"objects_trimmed":25,"snaptrim_duration":1.33035098,"stat_sum":{"num_bytes":2844902048,"num_objects":3045,"num_object_clones":2405,"num_object_copies":6090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3045,"num_whiteouts":0,"num_read":172364,"num_read_kb":601256,"num_write":149781,"num_write_kb":596273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"3"}]},{"pgid":"2.4","version":"315'177567","reported_seq":316185,"reported_epoch":315,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:08.956376+0000","last_change":"2023-10-03T18:23:05.853763+0000","last_active":"2023-10-03T18:23:08.956376+0000","last_peered":"2023-10-03T18:23:08.956376+0000","last_clean":"2023-10-03T18:23:08.956376+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:23:08.956376+0000","last_undegraded":"2023-10-03T18:23:08.956376+0000","last_fullsized":"2023-10-03T18:23:08.956376+0000","mapping_epoch":12,"log_start":"262'167473","ondisk_log_start":"262'167473","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10094,"log_dups_size":0,"ondisk_log_size":10094,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":177,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":3.4522545010000001,"stat_sum":{"num_bytes":2831082210,"num_objects":7757,"num_object_clones":7119,"num_object_copies":15514,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7757,"num_whiteouts":0,"num_read":160605,"num_read_kb":587845,"num_write":147794,"num_write_kb":589542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"139","length":"6"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23512466705,"num_objects":60251,"num_object_clones":54987,"num_object_copies":120502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60251,"num_whiteouts":0,"num_read":1345788,"num_read_kb":4899880,"num_write":1229947,"num_write_kb":4904425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10038009856,"data_stored":101729937954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80522,"ondisk_log_size":80522,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738973,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3834596,"kb_used_data":1884124,"kb_used_omap":1373,"kb_used_meta":1949090,"kb_avail":90537244,"statfs":{"total":96636764160,"available":92710137856,"internally_reserved":0,"allocated":1929342976,"data_stored":18626349973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1406319,"internal_metadata":1995868817},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,3,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":3.1960000000000002,"5min":2.5339999999999998,"15min":2.7090000000000001},"min":{"1min":0.23200000000000001,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":15.964,"5min":15.964,"15min":32.173000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":2.0030000000000001,"5min":2.3740000000000001,"15min":2.23},"min":{"1min":0.252,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":9.1769999999999996,"5min":30.128,"15min":30.128},"last":0.314}]},{"osd":1,"last update":"Tue Oct 3 18:22:16 2023","interfaces":[{"interface":"back","average":{"1min":2.1669999999999998,"5min":2.4990000000000001,"15min":2.5430000000000001},"min":{"1min":0.26900000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":12.064,"5min":17.274000000000001,"15min":18.768999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":3.0379999999999998,"5min":2.5129999999999999,"15min":2.4129999999999998},"min":{"1min":0.22900000000000001,"5min":0.214,"15min":0.188},"max":{"1min":10.451000000000001,"5min":12.346,"15min":18.751999999999999},"last":0.372}]}]},{"osd":0,"up_from":8,"seq":34359738972,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4126556,"kb_used_data":3023572,"kb_used_omap":2116,"kb_used_meta":1100859,"kb_avail":90245284,"statfs":{"total":96636764160,"available":92411170816,"internally_reserved":0,"allocated":3096137728,"data_stored":32099481592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1127280127},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":2.4009999999999998,"15min":3.097},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":5.9889999999999999,"5min":19.969999999999999,"15min":31.474},"last":5.9109999999999996},{"interface":"front","average":{"1min":2.883,"5min":3.2229999999999999,"15min":3.024},"min":{"1min":0.25900000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":10.037000000000001,"5min":19.498999999999999,"15min":30.273},"last":3.2130000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.331,"5min":1.806,"15min":1.8899999999999999},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":12.752000000000001,"5min":13.058,"15min":13.209},"last":5.5880000000000001},{"interface":"front","average":{"1min":2.621,"5min":2.4220000000000002,"15min":2.1469999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":13.459,"5min":13.459,"15min":16.033999999999999},"last":3.1789999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738949,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6606784,"kb_used_data":4904816,"kb_used_omap":3636,"kb_used_meta":1698315,"kb_avail":87765056,"statfs":{"total":96636764160,"available":89871417344,"internally_reserved":0,"allocated":5022531584,"data_stored":50788782244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":1739074667},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3,1,5,5,18,7,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:22:25 2023","interfaces":[{"interface":"back","average":{"1min":4.5090000000000003,"5min":4.2169999999999996,"15min":2.98},"min":{"1min":0.19800000000000001,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":30.896999999999998,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.5409999999999999,"5min":2.9199999999999999,"15min":2.6219999999999999},"min":{"1min":0.316,"5min":0.19900000000000001,"15min":0.17799999999999999},"max":{"1min":15.766999999999999,"5min":21.498999999999999,"15min":21.498999999999999},"last":5.0830000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:22:25 2023","interfaces":[{"interface":"back","average":{"1min":2.9430000000000001,"5min":2.3490000000000002,"15min":2.0579999999999998},"min":{"1min":0.23599999999999999,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":13.358000000000001,"5min":13.358000000000001,"15min":21.326000000000001},"last":1.024},{"interface":"front","average":{"1min":2.5409999999999999,"5min":2.5489999999999999,"15min":2.5880000000000001},"min":{"1min":0.29099999999999998,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":15.206,"5min":15.351000000000001,"15min":15.351000000000001},"last":5.1239999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3092627456,"data_stored":32177015909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5019254784,"data_stored":50893391121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1926127616,"data_stored":18659530924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:23:12.025 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:12.025 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:12.026 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:12.026 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:12.026 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:23:13.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:13.827+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:13.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:13.930+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:13.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:13.971+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:13.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:13.971+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:14.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.039+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:14.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.039+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:14.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.039+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:14.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.057+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:14.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.064+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:14.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.740+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:23:14.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.761+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:14.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.761+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:14.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.761+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:14.790 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.788+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:14.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.824+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:14.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.980+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:14.982 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.980+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:14.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:14.993+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:15.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.093+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:15.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.129+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:15.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.927+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:15.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.955+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:15.979 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.979+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:15.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:15.979+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:16.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.002+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:16.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.002+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:16.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.002+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:16.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.002+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:16.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.004+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:16.030 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:16.023+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:21.430 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.430+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:21.431 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.430+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:21.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.464+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:21.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.464+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:21.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.484+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:21.491 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.484+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:21.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.563+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:21.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.605+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:21.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:21.618+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:22.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.904+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:22.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.904+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:22.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.929+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:22.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.929+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:22.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.939+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:22.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.939+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:22.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.939+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:23.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:22.999+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:23.087 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:23.086+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:23.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:23.967+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:24.059 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.051+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:24.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.051+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:24.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.100+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:24.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.140+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:24.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.140+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:24.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.152+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:24.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.185+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:24.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.202+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:24.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:24.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:24.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.738+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:24.798 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.798+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:24.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.798+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:24.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.861+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:24.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.861+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:24.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.911+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:24.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.911+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:24.923 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:24.923+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:26.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.740+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:26.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.749+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:26.775 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.775+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:26.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.826+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:26.906 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.905+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:26.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.935+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:26.944 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.941+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:26.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:26.976+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:27.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.221+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:27.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.221+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:27.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.709+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:27.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.711+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:27.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.711+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:27.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.711+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:27.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:27.717+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:29.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.064+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:29.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.064+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:29.066 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.065+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:29.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.149+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:29.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.149+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:29.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.372+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:29.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.373+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:29.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.373+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:29.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.373+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:29.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:29.373+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:31.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:31.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:31.657+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:32.027 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:23:32.215 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.214+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:32.261 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.260+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:32.261 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.260+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:32.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.310+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:32.344 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.339+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:32.344 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.339+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:32.344 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.339+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:32.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.350+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:32.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:32.350+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:32.772 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:23:32.816 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:23:32.825 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1832,"stamp":"2023-10-03T18:23:32.357893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23534634801,"num_objects":61509,"num_object_clones":56243,"num_object_copies":123018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61509,"num_whiteouts":0,"num_read":1352776,"num_read_kb":4922815,"num_write":1235185,"num_write_kb":4926723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80435,"ondisk_log_size":80435,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13558944,"kb_used_data":9853640,"kb_used_omap":7129,"kb_used_meta":3698150,"kb_avail":269556576,"statfs":{"total":289910292480,"available":276025933824,"internally_reserved":0,"allocated":10090127360,"data_stored":104392300539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7300370,"internal_metadata":3786906350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,8,6,10,15,17,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11513856,"num_objects":394,"num_object_clones":394,"num_object_copies":788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":394,"num_whiteouts":0,"num_read":2908,"num_read_kb":11335,"num_write":2702,"num_write_kb":10808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.131048"},"pg_stats":[{"pgid":"2.7","version":"317'182572","reported_seq":325905,"reported_epoch":317,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:31.150691+0000","last_change":"2023-10-03T18:23:28.054452+0000","last_active":"2023-10-03T18:23:31.150691+0000","last_peered":"2023-10-03T18:23:31.150691+0000","last_clean":"2023-10-03T18:23:31.150691+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:23:31.150691+0000","last_undegraded":"2023-10-03T18:23:31.150691+0000","last_fullsized":"2023-10-03T18:23:31.150691+0000","mapping_epoch":12,"log_start":"268'172522","ondisk_log_start":"268'172522","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":253,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:00:10.053676+0000","scrub_duration":505.38444223300002,"objects_trimmed":69,"snaptrim_duration":10.679585255999999,"stat_sum":{"num_bytes":2931802647,"num_objects":10713,"num_object_clones":10061,"num_object_copies":21426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10713,"num_whiteouts":0,"num_read":165106,"num_read_kb":612418,"num_write":153448,"num_write_kb":612187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13b","length":"2"}]},{"pgid":"2.6","version":"317'169119","reported_seq":302252,"reported_epoch":317,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:31.090330+0000","last_change":"2023-10-03T18:23:29.235514+0000","last_active":"2023-10-03T18:23:31.090330+0000","last_peered":"2023-10-03T18:23:31.090330+0000","last_clean":"2023-10-03T18:23:31.090330+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:23:31.090330+0000","last_undegraded":"2023-10-03T18:23:31.090330+0000","last_fullsized":"2023-10-03T18:23:31.090330+0000","mapping_epoch":12,"log_start":"266'159080","ondisk_log_start":"266'159080","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":169,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":53,"snaptrim_duration":7.5623754029999999,"stat_sum":{"num_bytes":2708808252,"num_objects":7510,"num_object_clones":6907,"num_object_copies":15020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7510,"num_whiteouts":0,"num_read":153487,"num_read_kb":565681,"num_write":141795,"num_write_kb":566072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"1"}]},{"pgid":"2.5","version":"317'203213","reported_seq":360025,"reported_epoch":317,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:31.052507+0000","last_change":"2023-10-03T18:23:28.092521+0000","last_active":"2023-10-03T18:23:31.052507+0000","last_peered":"2023-10-03T18:23:31.052507+0000","last_clean":"2023-10-03T18:23:31.052507+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:23:31.052507+0000","last_undegraded":"2023-10-03T18:23:31.052507+0000","last_fullsized":"2023-10-03T18:23:31.052507+0000","mapping_epoch":12,"log_start":"272'193184","ondisk_log_start":"272'193184","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":196,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":76,"snaptrim_duration":6.609462433,"stat_sum":{"num_bytes":3177308816,"num_objects":9672,"num_object_clones":8962,"num_object_copies":19344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9672,"num_whiteouts":0,"num_read":188083,"num_read_kb":668651,"num_write":167163,"num_write_kb":665826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"4"}]},{"pgid":"2.0","version":"317'193506","reported_seq":342593,"reported_epoch":317,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:27.538586+0000","last_change":"2023-10-03T18:23:26.724213+0000","last_active":"2023-10-03T18:23:27.538586+0000","last_peered":"2023-10-03T18:23:27.538586+0000","last_clean":"2023-10-03T18:23:27.538586+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:23:27.538586+0000","last_undegraded":"2023-10-03T18:23:27.538586+0000","last_fullsized":"2023-10-03T18:23:27.538586+0000","mapping_epoch":12,"log_start":"285'183479","ondisk_log_start":"285'183479","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":224,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:04:44.882047+0000","scrub_duration":734.24538164700004,"objects_trimmed":141,"snaptrim_duration":9.3568578630000001,"stat_sum":{"num_bytes":3069166223,"num_objects":10245,"num_object_clones":9560,"num_object_copies":20490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10245,"num_whiteouts":0,"num_read":171703,"num_read_kb":637850,"num_write":161169,"num_write_kb":643111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"}]},{"pgid":"1.0","version":"88'99","reported_seq":479,"reported_epoch":316,"state":"active+clean","last_fresh":"2023-10-03T18:23:12.384252+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:23:12.384252+0000","last_peered":"2023-10-03T18:23:12.384252+0000","last_clean":"2023-10-03T18:23:12.384252+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:23:12.384252+0000","last_undegraded":"2023-10-03T18:23:12.384252+0000","last_fullsized":"2023-10-03T18:23:12.384252+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"317'195637","reported_seq":346468,"reported_epoch":317,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:31.085084+0000","last_change":"2023-10-03T18:23:29.231130+0000","last_active":"2023-10-03T18:23:31.085084+0000","last_peered":"2023-10-03T18:23:31.085084+0000","last_clean":"2023-10-03T18:23:31.085084+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:23:31.085084+0000","last_undegraded":"2023-10-03T18:23:31.085084+0000","last_fullsized":"2023-10-03T18:23:31.085084+0000","mapping_epoch":12,"log_start":"271'185593","ondisk_log_start":"271'185593","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":186,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":71,"snaptrim_duration":11.85711858,"stat_sum":{"num_bytes":3122029386,"num_objects":9241,"num_object_clones":8542,"num_object_copies":18482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9241,"num_whiteouts":0,"num_read":175055,"num_read_kb":649843,"num_write":163614,"num_write_kb":652811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"2"}]},{"pgid":"2.1","version":"317'190221","reported_seq":327345,"reported_epoch":317,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:27.585147+0000","last_change":"2023-10-03T18:23:27.284707+0000","last_active":"2023-10-03T18:23:27.585147+0000","last_peered":"2023-10-03T18:23:27.585147+0000","last_clean":"2023-10-03T18:23:27.585147+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:23:27.585147+0000","last_undegraded":"2023-10-03T18:23:27.585147+0000","last_fullsized":"2023-10-03T18:23:27.585147+0000","mapping_epoch":12,"log_start":"273'180163","ondisk_log_start":"273'180163","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:17:53.211569+0000","scrub_duration":424.08306342600002,"objects_trimmed":135,"snaptrim_duration":9.8554979389999993,"stat_sum":{"num_bytes":2843595475,"num_objects":3061,"num_object_clones":2424,"num_object_copies":6122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":164520,"num_read_kb":593600,"num_write":149017,"num_write_kb":593939,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"}]},{"pgid":"2.2","version":"317'190563","reported_seq":332408,"reported_epoch":317,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:28.912588+0000","last_change":"2023-10-03T18:23:27.639423+0000","last_active":"2023-10-03T18:23:28.912588+0000","last_peered":"2023-10-03T18:23:28.912588+0000","last_clean":"2023-10-03T18:23:28.912588+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:23:28.912588+0000","last_undegraded":"2023-10-03T18:23:28.912588+0000","last_fullsized":"2023-10-03T18:23:28.912588+0000","mapping_epoch":12,"log_start":"274'180507","ondisk_log_start":"274'180507","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:42:49.396697+0000","scrub_duration":370.574046178,"objects_trimmed":176,"snaptrim_duration":10.269565948,"stat_sum":{"num_bytes":2847425184,"num_objects":3064,"num_object_clones":2424,"num_object_copies":6128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3064,"num_whiteouts":0,"num_read":173260,"num_read_kb":603927,"num_write":150396,"num_write_kb":598733,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"}]},{"pgid":"2.4","version":"317'178710","reported_seq":317734,"reported_epoch":317,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:31.088687+0000","last_change":"2023-10-03T18:23:26.496045+0000","last_active":"2023-10-03T18:23:31.088687+0000","last_peered":"2023-10-03T18:23:31.088687+0000","last_clean":"2023-10-03T18:23:31.088687+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:23:31.088687+0000","last_undegraded":"2023-10-03T18:23:31.088687+0000","last_fullsized":"2023-10-03T18:23:31.088687+0000","mapping_epoch":12,"log_start":"268'168677","ondisk_log_start":"268'168677","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":174,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":9.1014226100000002,"stat_sum":{"num_bytes":2833908450,"num_objects":8001,"num_object_clones":7363,"num_object_copies":16002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8001,"num_whiteouts":0,"num_read":161456,"num_read_kb":590754,"num_write":148446,"num_write_kb":592150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13b","length":"4"},{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23534044433,"num_objects":61507,"num_object_clones":56243,"num_object_copies":123014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61507,"num_whiteouts":0,"num_read":1352670,"num_read_kb":4922724,"num_write":1235048,"num_write_kb":4924829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10079543296,"data_stored":104404896290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80336,"ondisk_log_size":80336,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738977,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2510260,"kb_used_data":1891564,"kb_used_omap":1375,"kb_used_meta":617312,"kb_avail":91861580,"statfs":{"total":96636764160,"available":94066257920,"internally_reserved":0,"allocated":1936961536,"data_stored":19079936991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":632127620},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,0,2,2,5,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":2.552,"15min":2.6389999999999998},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.1849999999999996,"5min":15.964,"15min":32.173000000000002},"last":4.9279999999999999},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.9650000000000001,"15min":2.153},"min":{"1min":0.28499999999999998,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":4.9260000000000002,"5min":10.586,"15min":30.128},"last":3.2450000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":2.2650000000000001,"15min":2.4820000000000002},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":5.6630000000000003,"5min":12.064,"15min":18.768999999999998},"last":1.603},{"interface":"front","average":{"1min":1.738,"5min":2.5590000000000002,"15min":2.3889999999999998},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":4.2919999999999998,"5min":12.346,"15min":18.751999999999999},"last":4.4029999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738976,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4257340,"kb_used_data":3035700,"kb_used_omap":2116,"kb_used_meta":1219515,"kb_avail":90114500,"statfs":{"total":96636764160,"available":92277248000,"internally_reserved":0,"allocated":3108556800,"data_stored":33111818680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1248783871},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,0,1,2,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.089,"5min":2.4009999999999998,"15min":3.097},"min":{"1min":0.186,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":5.9889999999999999,"5min":19.969999999999999,"15min":31.474},"last":2.9550000000000001},{"interface":"front","average":{"1min":2.883,"5min":3.2229999999999999,"15min":3.024},"min":{"1min":0.25900000000000001,"5min":0.20699999999999999,"15min":0.186},"max":{"1min":10.037000000000001,"5min":19.498999999999999,"15min":30.273},"last":0.309}]},{"osd":2,"last update":"Tue Oct 3 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":2.331,"5min":1.806,"15min":1.8899999999999999},"min":{"1min":0.26400000000000001,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":12.752000000000001,"5min":13.058,"15min":13.209},"last":0.27900000000000003},{"interface":"front","average":{"1min":2.621,"5min":2.4220000000000002,"15min":2.1469999999999998},"min":{"1min":0.28000000000000003,"5min":0.218,"15min":0.20999999999999999},"max":{"1min":13.459,"5min":13.459,"15min":16.033999999999999},"last":0.33800000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738953,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6791344,"kb_used_data":4926376,"kb_used_omap":3636,"kb_used_meta":1861323,"kb_avail":87580496,"statfs":{"total":96636764160,"available":89682427904,"internally_reserved":0,"allocated":5044609024,"data_stored":52200544868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":1905994859},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,6,8,12,10,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":3.1680000000000001,"5min":4.0730000000000004,"15min":3.056},"min":{"1min":0.184,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":22.068999999999999,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.27200000000000002},{"interface":"front","average":{"1min":2.79,"5min":2.806,"15min":2.625},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":13.994,"5min":19.434000000000001,"15min":21.498999999999999},"last":0.435}]},{"osd":2,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":2.3010000000000002,"15min":2.1019999999999999},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.19400000000000001},"max":{"1min":8.907,"5min":13.358000000000001,"15min":21.326000000000001},"last":6.9080000000000004},{"interface":"front","average":{"1min":3.0019999999999998,"5min":2.5939999999999999,"15min":2.645},"min":{"1min":0.25800000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":20.975999999999999,"5min":20.975999999999999,"15min":20.975999999999999},"last":5.9699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3104542720,"data_stored":33109199973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5041324032,"data_stored":52217132305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1933676544,"data_stored":19078564012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:23:32.826 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:32.827 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:32.827 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:32.827 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:32.827 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:23:34.502 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:34.499+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:23:34.584 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:34.583+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:23:34.610 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:23:34.609+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:23:35.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.006+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:35.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.038+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:35.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.038+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:35.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.038+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:35.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.038+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:35.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.038+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:35.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.626+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:35.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.626+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:35.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.626+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:35.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.714+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:35.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.714+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:35.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.714+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:35.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.714+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:35.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:35.714+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:37.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.173+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:37.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.234+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:37.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.234+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:37.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.234+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:37.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.234+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:37.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.267+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:37.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.267+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:37.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.267+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:37.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.267+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:37.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:37.268+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:38.943 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:23:38.941+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:23:44.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:44.580+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:44.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:44.580+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:44.628 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:44.626+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:44.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:44.669+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:44.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:44.669+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:45.441 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.435+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:45.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.450+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:45.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.550+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:45.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.550+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:45.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.550+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:45.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.551+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:45.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.640+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:45.729 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:45.721+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:46.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.175+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:46.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.175+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:46.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.175+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:46.177 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.175+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:46.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.200+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:46.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:46.201+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:49.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.100+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:49.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.100+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:49.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.236+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:49.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.278+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:49.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.278+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:49.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.319+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:49.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.325+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:49.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:49.348+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:50.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.429+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:50.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.429+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:50.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.446+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:50.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.497+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:50.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.537+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:50.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.636+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:50.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.636+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:50.646 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.643+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:50.648 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.647+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:50.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:50.647+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:51.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.678+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:23:51.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.681+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:23:51.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.681+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:51.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.796+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:51.831 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.829+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:51.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.833+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:51.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.841+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:51.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.841+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:51.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.841+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:51.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.841+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:51.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:51.841+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:52.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:23:53.614 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:23:53.623 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:23:53.637 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1846,"stamp":"2023-10-03T18:23:53.248843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23563654961,"num_objects":61180,"num_object_clones":55914,"num_object_copies":122360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61180,"num_whiteouts":0,"num_read":1359874,"num_read_kb":4951071,"num_write":1242115,"num_write_kb":4954443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80575,"ondisk_log_size":80575,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":13846424,"kb_used_data":9906492,"kb_used_omap":7129,"kb_used_meta":3932774,"kb_avail":269269096,"statfs":{"total":289910292480,"available":275731554304,"internally_reserved":0,"allocated":10144247808,"data_stored":104201687979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7300370,"internal_metadata":4027161326},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,9,4,6,8,6,13,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14106624,"num_objects":-82,"num_object_clones":-82,"num_object_copies":-164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-82,"num_whiteouts":0,"num_read":3531,"num_read_kb":14056,"num_write":3374,"num_write_kb":13496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001388"},"pg_stats":[{"pgid":"2.7","version":"321'183802","reported_seq":327831,"reported_epoch":321,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:52.582168+0000","last_change":"2023-10-03T18:23:51.147076+0000","last_active":"2023-10-03T18:23:52.582168+0000","last_peered":"2023-10-03T18:23:52.582168+0000","last_clean":"2023-10-03T18:23:52.582168+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:23:52.582168+0000","last_undegraded":"2023-10-03T18:23:52.582168+0000","last_fullsized":"2023-10-03T18:23:52.582168+0000","mapping_epoch":12,"log_start":"273'173724","ondisk_log_start":"273'173724","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":249,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:00:10.053676+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":3.7550295509999998,"stat_sum":{"num_bytes":2935165463,"num_objects":10709,"num_object_clones":10057,"num_object_copies":21418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10709,"num_whiteouts":0,"num_read":165959,"num_read_kb":615802,"num_write":154263,"num_write_kb":615447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"149","length":"1"}]},{"pgid":"2.6","version":"321'170322","reported_seq":304087,"reported_epoch":321,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:52.576955+0000","last_change":"2023-10-03T18:23:52.355291+0000","last_active":"2023-10-03T18:23:52.576955+0000","last_peered":"2023-10-03T18:23:52.576955+0000","last_clean":"2023-10-03T18:23:52.576955+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:23:52.576955+0000","last_undegraded":"2023-10-03T18:23:52.576955+0000","last_fullsized":"2023-10-03T18:23:52.576955+0000","mapping_epoch":12,"log_start":"270'160285","ondisk_log_start":"270'160285","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":164,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.9556987719999999,"stat_sum":{"num_bytes":2712228412,"num_objects":7535,"num_object_clones":6932,"num_object_copies":15070,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7535,"num_whiteouts":0,"num_read":154282,"num_read_kb":568849,"num_write":142630,"num_write_kb":569412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"4"}]},{"pgid":"2.5","version":"321'204479","reported_seq":362087,"reported_epoch":321,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:52.583716+0000","last_change":"2023-10-03T18:23:51.743103+0000","last_active":"2023-10-03T18:23:52.583716+0000","last_peered":"2023-10-03T18:23:52.583716+0000","last_clean":"2023-10-03T18:23:52.583716+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:23:52.583716+0000","last_undegraded":"2023-10-03T18:23:52.583716+0000","last_fullsized":"2023-10-03T18:23:52.583716+0000","mapping_epoch":12,"log_start":"281'194388","ondisk_log_start":"281'194388","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":192,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.91925990300000004,"stat_sum":{"num_bytes":3180905104,"num_objects":9697,"num_object_clones":8987,"num_object_copies":19394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9697,"num_whiteouts":0,"num_read":189024,"num_read_kb":672407,"num_write":168022,"num_write_kb":669262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"8"}]},{"pgid":"2.0","version":"321'195507","reported_seq":345358,"reported_epoch":321,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:52.765870+0000","last_change":"2023-10-03T18:23:50.524309+0000","last_active":"2023-10-03T18:23:52.765870+0000","last_peered":"2023-10-03T18:23:52.765870+0000","last_clean":"2023-10-03T18:23:52.765870+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:23:52.765870+0000","last_undegraded":"2023-10-03T18:23:52.765870+0000","last_fullsized":"2023-10-03T18:23:52.765870+0000","mapping_epoch":12,"log_start":"290'185490","ondisk_log_start":"290'185490","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10017,"log_dups_size":0,"ondisk_log_size":10017,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":212,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:04:44.882047+0000","scrub_duration":734.24538164700004,"objects_trimmed":4,"snaptrim_duration":3.1495684160000001,"stat_sum":{"num_bytes":3073454735,"num_objects":10057,"num_object_clones":9372,"num_object_copies":20114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10057,"num_whiteouts":0,"num_read":172786,"num_read_kb":642162,"num_write":162200,"num_write_kb":647235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":482,"reported_epoch":319,"state":"active+clean","last_fresh":"2023-10-03T18:23:42.409667+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:23:42.409667+0000","last_peered":"2023-10-03T18:23:42.409667+0000","last_clean":"2023-10-03T18:23:42.409667+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:23:42.409667+0000","last_undegraded":"2023-10-03T18:23:42.409667+0000","last_fullsized":"2023-10-03T18:23:42.409667+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"321'196946","reported_seq":348522,"reported_epoch":321,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:23:52.408767+0000","last_change":"2023-10-03T18:23:51.708215+0000","last_active":"2023-10-03T18:23:52.408767+0000","last_peered":"2023-10-03T18:23:52.408767+0000","last_clean":"2023-10-03T18:23:52.408767+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:23:52.408767+0000","last_undegraded":"2023-10-03T18:23:52.408767+0000","last_fullsized":"2023-10-03T18:23:52.408767+0000","mapping_epoch":12,"log_start":"278'186894","ondisk_log_start":"278'186894","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":182,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.3161436890000004,"stat_sum":{"num_bytes":3125609290,"num_objects":9235,"num_object_clones":8536,"num_object_copies":18470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9235,"num_whiteouts":0,"num_read":175973,"num_read_kb":653499,"num_write":164479,"num_write_kb":656271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"}]},{"pgid":"2.1","version":"321'191841","reported_seq":329717,"reported_epoch":321,"state":"active+clean","last_fresh":"2023-10-03T18:23:52.764981+0000","last_change":"2023-10-03T18:23:43.094561+0000","last_active":"2023-10-03T18:23:52.764981+0000","last_peered":"2023-10-03T18:23:52.764981+0000","last_clean":"2023-10-03T18:23:52.764981+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:23:52.764981+0000","last_undegraded":"2023-10-03T18:23:52.764981+0000","last_fullsized":"2023-10-03T18:23:52.764981+0000","mapping_epoch":12,"log_start":"285'181769","ondisk_log_start":"285'181769","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:17:53.211569+0000","scrub_duration":424.08306342600002,"objects_trimmed":13,"snaptrim_duration":0.70950967099999995,"stat_sum":{"num_bytes":2847662803,"num_objects":2975,"num_object_clones":2338,"num_object_copies":5950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2975,"num_whiteouts":0,"num_read":165488,"num_read_kb":597448,"num_write":149976,"num_write_kb":597775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"}]},{"pgid":"2.2","version":"320'191956","reported_seq":334320,"reported_epoch":320,"state":"active+clean","last_fresh":"2023-10-03T18:23:48.952390+0000","last_change":"2023-10-03T18:23:43.653472+0000","last_active":"2023-10-03T18:23:48.952390+0000","last_peered":"2023-10-03T18:23:48.952390+0000","last_clean":"2023-10-03T18:23:48.952390+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:23:48.952390+0000","last_undegraded":"2023-10-03T18:23:48.952390+0000","last_fullsized":"2023-10-03T18:23:48.952390+0000","mapping_epoch":12,"log_start":"285'181917","ondisk_log_start":"285'181917","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:42:49.396697+0000","scrub_duration":370.574046178,"objects_trimmed":19,"snaptrim_duration":1.266056418,"stat_sum":{"num_bytes":2850845344,"num_objects":2960,"num_object_clones":2320,"num_object_copies":5920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2960,"num_whiteouts":0,"num_read":173967,"num_read_kb":606739,"num_write":151178,"num_write_kb":601861,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"148","length":"a"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"}]},{"pgid":"2.4","version":"321'179868","reported_seq":319584,"reported_epoch":321,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:23:52.521980+0000","last_change":"2023-10-03T18:23:52.358853+0000","last_active":"2023-10-03T18:23:52.521980+0000","last_peered":"2023-10-03T18:23:52.521980+0000","last_clean":"2023-10-03T18:23:52.521980+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:23:52.521980+0000","last_undegraded":"2023-10-03T18:23:52.521980+0000","last_fullsized":"2023-10-03T18:23:52.521980+0000","mapping_epoch":12,"log_start":"272'169778","ondisk_log_start":"272'169778","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":170,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":1.656795529,"stat_sum":{"num_bytes":2837193442,"num_objects":8010,"num_object_clones":7372,"num_object_copies":16020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8010,"num_whiteouts":0,"num_read":162289,"num_read_kb":594074,"num_write":149230,"num_write_kb":595286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"149","length":"9"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23563064593,"num_objects":61178,"num_object_clones":55914,"num_object_copies":122356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61178,"num_whiteouts":0,"num_read":1359768,"num_read_kb":4950980,"num_write":1241978,"num_write_kb":4952549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10134515712,"data_stored":104195398178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80476,"ondisk_log_size":80476,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738982,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2549936,"kb_used_data":1903464,"kb_used_omap":1375,"kb_used_meta":645088,"kb_avail":91821904,"statfs":{"total":96636764160,"available":94025629696,"internally_reserved":0,"allocated":1949147136,"data_stored":18912205423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":660570244},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,2,3,3,4,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":2.552,"15min":2.6389999999999998},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.1849999999999996,"5min":15.964,"15min":32.173000000000002},"last":2.198},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.9650000000000001,"15min":2.153},"min":{"1min":0.28499999999999998,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":4.9260000000000002,"5min":10.586,"15min":30.128},"last":2.895}]},{"osd":1,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":2.2650000000000001,"15min":2.4820000000000002},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":5.6630000000000003,"5min":12.064,"15min":18.768999999999998},"last":2.8639999999999999},{"interface":"front","average":{"1min":1.738,"5min":2.5590000000000002,"15min":2.3889999999999998},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":4.2919999999999998,"5min":12.346,"15min":18.751999999999999},"last":2.2160000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738980,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4340148,"kb_used_data":3050476,"kb_used_omap":2116,"kb_used_meta":1287547,"kb_avail":90031692,"statfs":{"total":96636764160,"available":92192452608,"internally_reserved":0,"allocated":3123687424,"data_stored":33177440328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1318448639},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":2.3849999999999998,"15min":3.052},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":8.8450000000000006,"5min":19.969999999999999,"15min":31.474},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.839,"15min":3.0419999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":8.9760000000000009,"5min":10.037000000000001,"15min":30.273},"last":0.222}]},{"osd":2,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.893,"15min":1.9410000000000001},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":8.0709999999999997,"5min":13.058,"15min":13.209},"last":0.35999999999999999},{"interface":"front","average":{"1min":2.2730000000000001,"5min":2.4710000000000001,"15min":2.1629999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":8.2050000000000001,"5min":13.459,"15min":16.033999999999999},"last":0.34399999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738957,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6956340,"kb_used_data":4952552,"kb_used_omap":3636,"kb_used_meta":2000139,"kb_avail":87415500,"statfs":{"total":96636764160,"available":89513472000,"internally_reserved":0,"allocated":5071413248,"data_stored":52112042228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":2048142443},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,3,2,2,3,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":3.1680000000000001,"5min":4.0730000000000004,"15min":3.056},"min":{"1min":0.184,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":22.068999999999999,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.51200000000000001},{"interface":"front","average":{"1min":2.79,"5min":2.806,"15min":2.625},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":13.994,"5min":19.434000000000001,"15min":21.498999999999999},"last":4.6769999999999996}]},{"osd":2,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":2.3010000000000002,"15min":2.1019999999999999},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.19400000000000001},"max":{"1min":8.907,"5min":13.358000000000001,"15min":21.326000000000001},"last":6.173},{"interface":"front","average":{"1min":3.0019999999999998,"5min":2.5939999999999999,"15min":2.645},"min":{"1min":0.25800000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":20.975999999999999,"5min":20.975999999999999,"15min":20.975999999999999},"last":4.6390000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3119722496,"data_stored":33176198245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5068877824,"data_stored":52106192145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1945915392,"data_stored":18913007788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:23:53.639 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:53.639 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:53.639 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:53.639 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:23:53.640 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:23:54.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.806+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:54.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.806+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:54.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.806+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:54.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.806+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:54.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.810+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:54.819 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:54.817+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:56.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.191+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:56.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.191+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:56.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.225+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:56.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.225+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:56.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.231+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:56.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.305+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:56.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.326+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:56.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.686+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:23:56.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.708+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:23:56.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.714+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:56.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.714+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:56.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.777+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:23:56.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.777+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:23:56.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.778+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:56.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.778+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:56.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:56.792+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:23:57.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:57.310+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:23:57.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:57.328+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:23:57.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:57.357+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:23:57.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:57.367+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:23:57.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:23:57.383+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:00.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.129+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:00.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.129+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:00.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.130+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:00.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.130+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:00.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.165+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:00.182 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.165+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:00.182 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.165+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:00.182 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.165+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:00.263 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.262+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:00.280 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:00.278+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:02.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.025+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:02.028 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.025+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:02.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.124+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:02.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.124+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:02.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.124+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:02.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:02.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:02.227+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:04.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:04.178+0000 7fc515f98700 -1 reset not still connected to 0x562226b529c0 2023-10-03T18:24:05.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.303+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:05.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.303+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:05.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.501+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:05.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.540+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:05.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.548+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:05.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.548+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:05.568 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.567+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:05.616 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.614+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:05.684 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:05.683+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:06.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.225+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:06.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:06.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.450+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:06.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.450+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:06.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.460+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:06.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.512+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:06.513 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.512+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:06.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.663+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:06.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.709+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:06.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:06.987+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:07.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.124+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:07.144 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.144+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:07.145 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.144+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:07.147 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.147+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:07.183 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.183+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:07.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.234+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:07.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.234+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:07.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.252+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:07.276 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.274+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:07.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.304+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:07.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.304+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:07.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.337+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:07.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.337+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:07.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.350+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:07.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.372+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:07.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.385+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:07.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.447+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:07.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.447+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:07.478 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.478+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:07.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.495+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:07.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:07.508+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:08.050 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.043+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:08.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.231+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:08.344 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.340+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:08.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.356+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:08.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.380+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:08.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.380+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:08.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.453+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:08.504 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.503+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:08.504 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:08.503+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:09.906 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:09.904+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:09.999 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:09.918+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:09.999 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:09.983+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:10.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:09.987+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:10.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:10.040+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:10.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:10.040+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:10.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:10.065+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:10.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:10.249+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:10.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:10.249+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:13.171 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:13.170+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:24:13.171 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:13.170+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:24:13.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:13.497 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:13.494+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:13.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:24:14.339 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:24:14.348 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:24:14.368 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1865,"stamp":"2023-10-03T18:24:13.253015+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23585462065,"num_objects":61870,"num_object_clones":56604,"num_object_copies":123740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61870,"num_whiteouts":0,"num_read":1368297,"num_read_kb":4975023,"num_write":1247557,"num_write_kb":4975491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80595,"ondisk_log_size":80595,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14168208,"kb_used_data":9950768,"kb_used_omap":7129,"kb_used_meta":4210278,"kb_avail":268947312,"statfs":{"total":289910292480,"available":275402047488,"internally_reserved":0,"allocated":10189586432,"data_stored":105646017555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7300370,"internal_metadata":4311325422},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,2,5,5,31,15,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6291405,"num_objects":731,"num_object_clones":732,"num_object_copies":1462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":731,"num_whiteouts":0,"num_read":2170,"num_read_kb":6843,"num_write":1562,"num_write_kb":6052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.810968"},"pg_stats":[{"pgid":"2.7","version":"330'184814","reported_seq":329353,"reported_epoch":330,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:09.555196+0000","last_change":"2023-10-03T18:24:06.291824+0000","last_active":"2023-10-03T18:24:09.555196+0000","last_peered":"2023-10-03T18:24:09.555196+0000","last_clean":"2023-10-03T18:24:09.555196+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:24:09.555196+0000","last_undegraded":"2023-10-03T18:24:09.555196+0000","last_fullsized":"2023-10-03T18:24:09.555196+0000","mapping_epoch":12,"log_start":"281'174726","ondisk_log_start":"281'174726","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":267,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:15:24.143759+0000","scrub_duration":505.38444223300002,"objects_trimmed":10,"snaptrim_duration":0.23640846200000001,"stat_sum":{"num_bytes":2937622962,"num_objects":10827,"num_object_clones":10177,"num_object_copies":21654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10827,"num_whiteouts":0,"num_read":166905,"num_read_kb":618655,"num_write":154880,"num_write_kb":617842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"330'171244","reported_seq":305557,"reported_epoch":330,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:09.570430+0000","last_change":"2023-10-03T18:24:03.324101+0000","last_active":"2023-10-03T18:24:09.570430+0000","last_peered":"2023-10-03T18:24:09.570430+0000","last_clean":"2023-10-03T18:24:09.570430+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:24:09.570430+0000","last_undegraded":"2023-10-03T18:24:09.570430+0000","last_fullsized":"2023-10-03T18:24:09.570430+0000","mapping_epoch":12,"log_start":"274'161194","ondisk_log_start":"274'161194","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":182,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.84204893000000003,"stat_sum":{"num_bytes":2714649148,"num_objects":7649,"num_object_clones":7046,"num_object_copies":15298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7649,"num_whiteouts":0,"num_read":155115,"num_read_kb":571477,"num_write":143230,"num_write_kb":571776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"}]},{"pgid":"2.5","version":"330'205633","reported_seq":363822,"reported_epoch":330,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:09.574046+0000","last_change":"2023-10-03T18:24:02.764268+0000","last_active":"2023-10-03T18:24:09.574046+0000","last_peered":"2023-10-03T18:24:09.574046+0000","last_clean":"2023-10-03T18:24:09.574046+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:24:09.574046+0000","last_undegraded":"2023-10-03T18:24:09.574046+0000","last_fullsized":"2023-10-03T18:24:09.574046+0000","mapping_epoch":12,"log_start":"287'195591","ondisk_log_start":"287'195591","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":210,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":0.678194827,"stat_sum":{"num_bytes":3183723100,"num_objects":9841,"num_object_clones":9132,"num_object_copies":19682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9841,"num_whiteouts":0,"num_read":190264,"num_read_kb":675519,"num_write":168746,"num_write_kb":672017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"}]},{"pgid":"2.0","version":"330'197091","reported_seq":347369,"reported_epoch":330,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:12.820015+0000","last_change":"2023-10-03T18:24:12.411439+0000","last_active":"2023-10-03T18:24:12.820015+0000","last_peered":"2023-10-03T18:24:12.820015+0000","last_clean":"2023-10-03T18:24:12.820015+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:24:12.820015+0000","last_undegraded":"2023-10-03T18:24:12.820015+0000","last_fullsized":"2023-10-03T18:24:12.820015+0000","mapping_epoch":12,"log_start":"296'187001","ondisk_log_start":"296'187001","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":223,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:30:12.536337+0000","scrub_duration":734.24538164700004,"objects_trimmed":68,"snaptrim_duration":5.0024482929999996,"stat_sum":{"num_bytes":3076465447,"num_objects":10011,"num_object_clones":9323,"num_object_copies":20022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10011,"num_whiteouts":0,"num_read":173762,"num_read_kb":645141,"num_write":162945,"num_write_kb":650141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":493,"reported_epoch":329,"state":"active+clean","last_fresh":"2023-10-03T18:24:06.547252+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:24:06.547252+0000","last_peered":"2023-10-03T18:24:06.547252+0000","last_clean":"2023-10-03T18:24:06.547252+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:24:06.547252+0000","last_undegraded":"2023-10-03T18:24:06.547252+0000","last_fullsized":"2023-10-03T18:24:06.547252+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"330'197992","reported_seq":350158,"reported_epoch":330,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:09.363381+0000","last_change":"2023-10-03T18:24:02.742282+0000","last_active":"2023-10-03T18:24:09.363381+0000","last_peered":"2023-10-03T18:24:09.363381+0000","last_clean":"2023-10-03T18:24:09.363381+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:24:09.363381+0000","last_undegraded":"2023-10-03T18:24:09.363381+0000","last_fullsized":"2023-10-03T18:24:09.363381+0000","mapping_epoch":12,"log_start":"286'187899","ondisk_log_start":"286'187899","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":200,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.33937708500000002,"stat_sum":{"num_bytes":3128353610,"num_objects":9353,"num_object_clones":8654,"num_object_copies":18706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9353,"num_whiteouts":0,"num_read":176971,"num_read_kb":656536,"num_write":165166,"num_write_kb":658940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"}]},{"pgid":"2.1","version":"330'193110","reported_seq":331501,"reported_epoch":330,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:12.828196+0000","last_change":"2023-10-03T18:24:10.071294+0000","last_active":"2023-10-03T18:24:12.828196+0000","last_peered":"2023-10-03T18:24:12.828196+0000","last_clean":"2023-10-03T18:24:12.828196+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:24:12.828196+0000","last_undegraded":"2023-10-03T18:24:12.828196+0000","last_fullsized":"2023-10-03T18:24:12.828196+0000","mapping_epoch":12,"log_start":"288'183071","ondisk_log_start":"288'183071","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:16.974230+0000","scrub_duration":424.08306342600002,"objects_trimmed":72,"snaptrim_duration":2.6716534169999999,"stat_sum":{"num_bytes":2850517563,"num_objects":3063,"num_object_clones":2429,"num_object_copies":6126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3063,"num_whiteouts":0,"num_read":166609,"num_read_kb":600546,"num_write":150693,"num_write_kb":600530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"4"}]},{"pgid":"2.2","version":"330'193100","reported_seq":336224,"reported_epoch":330,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:09.138041+0000","last_change":"2023-10-03T18:24:07.930482+0000","last_active":"2023-10-03T18:24:09.138041+0000","last_peered":"2023-10-03T18:24:09.138041+0000","last_clean":"2023-10-03T18:24:09.138041+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:24:09.138041+0000","last_undegraded":"2023-10-03T18:24:09.138041+0000","last_fullsized":"2023-10-03T18:24:09.138041+0000","mapping_epoch":12,"log_start":"288'183022","ondisk_log_start":"288'183022","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:02:56.737492+0000","scrub_duration":370.574046178,"objects_trimmed":27,"snaptrim_duration":0.54031935499999995,"stat_sum":{"num_bytes":2853901061,"num_objects":3024,"num_object_clones":2382,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":175341,"num_read_kb":610303,"num_write":151938,"num_write_kb":604779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"3"}]},{"pgid":"2.4","version":"330'180798","reported_seq":321065,"reported_epoch":330,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:09.425776+0000","last_change":"2023-10-03T18:24:06.273354+0000","last_active":"2023-10-03T18:24:09.425776+0000","last_peered":"2023-10-03T18:24:09.425776+0000","last_clean":"2023-10-03T18:24:09.425776+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:24:09.425776+0000","last_undegraded":"2023-10-03T18:24:09.425776+0000","last_fullsized":"2023-10-03T18:24:09.425776+0000","mapping_epoch":12,"log_start":"278'170782","ondisk_log_start":"278'170782","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":187,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.80581425200000001,"stat_sum":{"num_bytes":2839638806,"num_objects":8100,"num_object_clones":7461,"num_object_copies":16200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8100,"num_whiteouts":0,"num_read":163224,"num_read_kb":596755,"num_write":149822,"num_write_kb":597572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14c","length":"6"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23584871697,"num_objects":61868,"num_object_clones":56604,"num_object_copies":123736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61868,"num_whiteouts":0,"num_read":1368191,"num_read_kb":4974932,"num_write":1247420,"num_write_kb":4973597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10178748416,"data_stored":105790491170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80496,"ondisk_log_size":80496,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738986,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2643276,"kb_used_data":1912196,"kb_used_omap":1375,"kb_used_meta":729696,"kb_avail":91728564,"statfs":{"total":96636764160,"available":93930049536,"internally_reserved":0,"allocated":1958088704,"data_stored":19168659431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":747208836},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,1,0,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.6539999999999999,"5min":2.552,"15min":2.6389999999999998},"min":{"1min":0.222,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":6.1849999999999996,"5min":15.964,"15min":32.173000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.9650000000000001,"15min":2.153},"min":{"1min":0.28499999999999998,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":4.9260000000000002,"5min":10.586,"15min":30.128},"last":0.91600000000000004}]},{"osd":1,"last update":"Tue Oct 3 18:23:17 2023","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":2.2650000000000001,"15min":2.4820000000000002},"min":{"1min":0.27600000000000002,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":5.6630000000000003,"5min":12.064,"15min":18.768999999999998},"last":1.649},{"interface":"front","average":{"1min":1.738,"5min":2.5590000000000002,"15min":2.3889999999999998},"min":{"1min":0.27600000000000002,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":4.2919999999999998,"5min":12.346,"15min":18.751999999999999},"last":0.307}]}]},{"osd":0,"up_from":8,"seq":34359738984,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4443940,"kb_used_data":3065492,"kb_used_omap":2116,"kb_used_meta":1376315,"kb_avail":89927900,"statfs":{"total":96636764160,"available":92086169600,"internally_reserved":0,"allocated":3139063808,"data_stored":33698962880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1409347071},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,1,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":2.3849999999999998,"15min":3.052},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":8.8450000000000006,"5min":19.969999999999999,"15min":31.474},"last":5.2880000000000003},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.839,"15min":3.0419999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":8.9760000000000009,"5min":10.037000000000001,"15min":30.273},"last":5.274}]},{"osd":2,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.893,"15min":1.9410000000000001},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":8.0709999999999997,"5min":13.058,"15min":13.209},"last":0.308},{"interface":"front","average":{"1min":2.2730000000000001,"5min":2.4710000000000001,"15min":2.1629999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":8.2050000000000001,"5min":13.459,"15min":16.033999999999999},"last":3.48}]}]},{"osd":1,"up_from":8,"seq":34359738960,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7080992,"kb_used_data":4973080,"kb_used_omap":3636,"kb_used_meta":2104267,"kb_avail":87290848,"statfs":{"total":96636764160,"available":89385828352,"internally_reserved":0,"allocated":5092433920,"data_stored":52778395244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":2154769515},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,1,3,31,10,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":3.1680000000000001,"5min":4.0730000000000004,"15min":3.056},"min":{"1min":0.184,"5min":0.182,"15min":0.17599999999999999},"max":{"1min":22.068999999999999,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.245},{"interface":"front","average":{"1min":2.79,"5min":2.806,"15min":2.625},"min":{"1min":0.23100000000000001,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":13.994,"5min":19.434000000000001,"15min":21.498999999999999},"last":1.3160000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":1.784,"5min":2.3010000000000002,"15min":2.1019999999999999},"min":{"1min":0.23599999999999999,"5min":0.22700000000000001,"15min":0.19400000000000001},"max":{"1min":8.907,"5min":13.358000000000001,"15min":21.326000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":3.0019999999999998,"5min":2.5939999999999999,"15min":2.645},"min":{"1min":0.25800000000000001,"5min":0.20799999999999999,"15min":0.20699999999999999},"max":{"1min":20.975999999999999,"5min":20.975999999999999,"15min":20.975999999999999},"last":1.353}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3135135744,"data_stored":33739029605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5088878592,"data_stored":52866422033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1954734080,"data_stored":19185039532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:24:14.369 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:14.370 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:14.370 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:14.370 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:14.370 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:24:14.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.575+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:14.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.575+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:14.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.575+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:14.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.575+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:14.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.583+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:14.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.583+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:14.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.583+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:14.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:14.583+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:15.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.063+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:15.175 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.174+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:15.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.185+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:15.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.185+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:15.189 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.185+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:15.424 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.422+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:15.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.455+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:15.462 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.455+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:15.468 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.468+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:15.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.530+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:15.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.575+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:15.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.654+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:15.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.749+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:15.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:15.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.824+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:15.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.909+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:15.910 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.910+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:15.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:15.955+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:16.004 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:16.004+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:16.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:16.037+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:18.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.353+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:18.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.355+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:18.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.355+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:18.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.455+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:18.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.455+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:18.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.455+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:18.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.500+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:18.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:18.503+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:19.926 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:19.925+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:24:19.927 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:19.925+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:24:19.927 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:19.925+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:24:21.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.658+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:21.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.671+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:21.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.673+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:21.721 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.717+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:21.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.764+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:21.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.764+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:21.772 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.772+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:21.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.800+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:21.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:21.800+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:25.295 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.293+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:25.295 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.293+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:25.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.358+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:25.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.358+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:25.417 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.413+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:25.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.525+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:25.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:25.532+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:28.694 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:28.694+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:28.717 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:28.717+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:28.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:28.717+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:28.976 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:28.976+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:28.977 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:28.976+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:29.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.032+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:29.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.032+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:29.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.032+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:29.100 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.099+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:29.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.188+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:29.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.288+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:29.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.314+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:29.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.314+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:29.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.663+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:29.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:29.681+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:30.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.354+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:30.402 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.399+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:30.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.529+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:30.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.529+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:30.533 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.529+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:30.533 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.529+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:30.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.534+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:30.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.553+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:30.554 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.553+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:30.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:30.983+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:31.026 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:31.025+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:31.037 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:31.036+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:31.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:31.036+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:31.038 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:31.036+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:31.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:31.112+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:32.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.044+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:32.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.063+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:32.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.063+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:32.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.063+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:32.126 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.125+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.142+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.142+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.142+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:32.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.142+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:32.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.227+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:32.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:32.227+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:33.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.224+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:33.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.224+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:33.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.224+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:33.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.240+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:33.264 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.264+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:33.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.290+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:33.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.290+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:33.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.290+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:33.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.339+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:33.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.394+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:33.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.636+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:33.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.638+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:33.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.780+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:33.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.780+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:33.830 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:33.831 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:33.829+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:34.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.028+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:34.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.079+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:34.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.079+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:34.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.183+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:34.184 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.183+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:34.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.193+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:34.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.193+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:34.195 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.193+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:34.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.255+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:34.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:34.255+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:34.371 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:24:35.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.001+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:35.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.052+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:35.069 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:24:35.077 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:24:35.093 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1878,"stamp":"2023-10-03T18:24:33.262105+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23607891761,"num_objects":62746,"num_object_clones":57480,"num_object_copies":125492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62746,"num_whiteouts":0,"num_read":1375172,"num_read_kb":4997858,"num_write":1252790,"num_write_kb":4996423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80424,"ondisk_log_size":80424,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14504836,"kb_used_data":9992424,"kb_used_omap":7129,"kb_used_meta":4505254,"kb_avail":268610684,"statfs":{"total":289910292480,"available":275057340416,"internally_reserved":0,"allocated":10232242176,"data_stored":107862715187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7300370,"internal_metadata":4613380846},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,9,5,17,10,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12009472,"num_objects":172,"num_object_clones":172,"num_object_copies":344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":4143,"num_read_kb":12528,"num_write":2780,"num_write_kb":11120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001840"},"pg_stats":[{"pgid":"2.7","version":"332'186017","reported_seq":330999,"reported_epoch":332,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:32.042994+0000","last_change":"2023-10-03T18:24:31.269603+0000","last_active":"2023-10-03T18:24:32.042994+0000","last_peered":"2023-10-03T18:24:32.042994+0000","last_clean":"2023-10-03T18:24:32.042994+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:24:32.042994+0000","last_undegraded":"2023-10-03T18:24:32.042994+0000","last_fullsized":"2023-10-03T18:24:32.042994+0000","mapping_epoch":12,"log_start":"287'175931","ondisk_log_start":"287'175931","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":263,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:15:24.143759+0000","scrub_duration":505.38444223300002,"objects_trimmed":48,"snaptrim_duration":5.7651041339999995,"stat_sum":{"num_bytes":2940567986,"num_objects":11027,"num_object_clones":10377,"num_object_copies":22054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11027,"num_whiteouts":0,"num_read":167775,"num_read_kb":621695,"num_write":155556,"num_write_kb":620546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14e","length":"2"}]},{"pgid":"2.6","version":"332'172420","reported_seq":307152,"reported_epoch":332,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:31.764122+0000","last_change":"2023-10-03T18:24:31.269514+0000","last_active":"2023-10-03T18:24:31.764122+0000","last_peered":"2023-10-03T18:24:31.764122+0000","last_clean":"2023-10-03T18:24:31.764122+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:24:31.764122+0000","last_undegraded":"2023-10-03T18:24:31.764122+0000","last_fullsized":"2023-10-03T18:24:31.764122+0000","mapping_epoch":12,"log_start":"285'162399","ondisk_log_start":"285'162399","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":178,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":51,"snaptrim_duration":8.8553713829999996,"stat_sum":{"num_bytes":2717352508,"num_objects":7794,"num_object_clones":7191,"num_object_copies":15588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7794,"num_whiteouts":0,"num_read":155933,"num_read_kb":574278,"num_write":143884,"num_write_kb":574392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14e","length":"4"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"2"}]},{"pgid":"2.5","version":"332'206953","reported_seq":365631,"reported_epoch":332,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:31.725293+0000","last_change":"2023-10-03T18:24:30.731886+0000","last_active":"2023-10-03T18:24:31.725293+0000","last_peered":"2023-10-03T18:24:31.725293+0000","last_clean":"2023-10-03T18:24:31.725293+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:24:31.725293+0000","last_undegraded":"2023-10-03T18:24:31.725293+0000","last_fullsized":"2023-10-03T18:24:31.725293+0000","mapping_epoch":12,"log_start":"291'196895","ondisk_log_start":"291'196895","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":206,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":68,"snaptrim_duration":8.3177467450000009,"stat_sum":{"num_bytes":3186831964,"num_objects":10060,"num_object_clones":9351,"num_object_copies":20120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10060,"num_whiteouts":0,"num_read":191312,"num_read_kb":678851,"num_write":169483,"num_write_kb":674965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14e","length":"4"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"2"}]},{"pgid":"2.0","version":"333'198548","reported_seq":349166,"reported_epoch":333,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:32.899258+0000","last_change":"2023-10-03T18:24:31.234702+0000","last_active":"2023-10-03T18:24:32.899258+0000","last_peered":"2023-10-03T18:24:32.899258+0000","last_clean":"2023-10-03T18:24:32.899258+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:24:32.899258+0000","last_undegraded":"2023-10-03T18:24:32.899258+0000","last_fullsized":"2023-10-03T18:24:32.899258+0000","mapping_epoch":12,"log_start":"304'188510","ondisk_log_start":"304'188510","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":214,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:30:12.536337+0000","scrub_duration":734.24538164700004,"objects_trimmed":8,"snaptrim_duration":8.8011897250000004,"stat_sum":{"num_bytes":3079111463,"num_objects":9990,"num_object_clones":9302,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9990,"num_whiteouts":0,"num_read":174568,"num_read_kb":647945,"num_write":163552,"num_write_kb":652569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14f","length":"3"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":495,"reported_epoch":331,"state":"active+clean","last_fresh":"2023-10-03T18:24:17.536901+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:24:17.536901+0000","last_peered":"2023-10-03T18:24:17.536901+0000","last_clean":"2023-10-03T18:24:17.536901+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:24:17.536901+0000","last_undegraded":"2023-10-03T18:24:17.536901+0000","last_fullsized":"2023-10-03T18:24:17.536901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"332'199309","reported_seq":351836,"reported_epoch":332,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:31.640983+0000","last_change":"2023-10-03T18:24:30.731989+0000","last_active":"2023-10-03T18:24:31.640983+0000","last_peered":"2023-10-03T18:24:31.640983+0000","last_clean":"2023-10-03T18:24:31.640983+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:24:31.640983+0000","last_undegraded":"2023-10-03T18:24:31.640983+0000","last_fullsized":"2023-10-03T18:24:31.640983+0000","mapping_epoch":12,"log_start":"290'189303","ondisk_log_start":"290'189303","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":197,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":45,"snaptrim_duration":3.9261791640000001,"stat_sum":{"num_bytes":3131548490,"num_objects":9560,"num_object_clones":8861,"num_object_copies":19120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9560,"num_whiteouts":0,"num_read":177806,"num_read_kb":659444,"num_write":165888,"num_write_kb":661828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14e","length":"4"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"3"}]},{"pgid":"2.1","version":"333'194602","reported_seq":333294,"reported_epoch":333,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:32.918849+0000","last_change":"2023-10-03T18:24:30.547697+0000","last_active":"2023-10-03T18:24:32.918849+0000","last_peered":"2023-10-03T18:24:32.918849+0000","last_clean":"2023-10-03T18:24:32.918849+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:24:32.918849+0000","last_undegraded":"2023-10-03T18:24:32.918849+0000","last_fullsized":"2023-10-03T18:24:32.918849+0000","mapping_epoch":12,"log_start":"296'184579","ondisk_log_start":"296'184579","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:16.974230+0000","scrub_duration":424.08306342600002,"objects_trimmed":8,"snaptrim_duration":8.1145750640000003,"stat_sum":{"num_bytes":2853245499,"num_objects":3000,"num_object_clones":2366,"num_object_copies":6000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3000,"num_whiteouts":0,"num_read":167401,"num_read_kb":603080,"num_write":151313,"num_write_kb":603010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14f","length":"3"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"}]},{"pgid":"2.2","version":"332'194545","reported_seq":337988,"reported_epoch":332,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:29.290092+0000","last_change":"2023-10-03T18:24:27.509314+0000","last_active":"2023-10-03T18:24:29.290092+0000","last_peered":"2023-10-03T18:24:29.290092+0000","last_clean":"2023-10-03T18:24:29.290092+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:24:29.290092+0000","last_undegraded":"2023-10-03T18:24:29.290092+0000","last_fullsized":"2023-10-03T18:24:29.290092+0000","mapping_epoch":12,"log_start":"293'184533","ondisk_log_start":"293'184533","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:02:56.737492+0000","scrub_duration":370.574046178,"objects_trimmed":93,"snaptrim_duration":5.0956396440000002,"stat_sum":{"num_bytes":2856305413,"num_objects":3014,"num_object_clones":2372,"num_object_copies":6028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3014,"num_whiteouts":0,"num_read":176239,"num_read_kb":612982,"num_write":152505,"num_write_kb":607047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"14e","length":"4"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"2"}]},{"pgid":"2.4","version":"332'181968","reported_seq":322610,"reported_epoch":332,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:31.739361+0000","last_change":"2023-10-03T18:24:29.055713+0000","last_active":"2023-10-03T18:24:31.739361+0000","last_peered":"2023-10-03T18:24:31.739361+0000","last_clean":"2023-10-03T18:24:31.739361+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:24:31.739361+0000","last_undegraded":"2023-10-03T18:24:31.739361+0000","last_fullsized":"2023-10-03T18:24:31.739361+0000","mapping_epoch":12,"log_start":"286'171887","ondisk_log_start":"286'171887","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":184,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":32,"snaptrim_duration":6.6401030050000003,"stat_sum":{"num_bytes":2842338070,"num_objects":8299,"num_object_clones":7660,"num_object_copies":16598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8299,"num_whiteouts":0,"num_read":164032,"num_read_kb":599492,"num_write":150472,"num_write_kb":600172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14e","length":"4"},{"start":"15a","length":"1"},{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23607301393,"num_objects":62744,"num_object_clones":57480,"num_object_copies":125488,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62744,"num_whiteouts":0,"num_read":1375066,"num_read_kb":4997767,"num_write":1252653,"num_write_kb":4994529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10221305856,"data_stored":107859462690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80325,"ondisk_log_size":80325,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738990,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2714636,"kb_used_data":1919744,"kb_used_omap":1375,"kb_used_meta":793504,"kb_avail":91657204,"statfs":{"total":96636764160,"available":93856976896,"internally_reserved":0,"allocated":1965817856,"data_stored":19356693063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":812548228},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.375,"5min":2.3650000000000002,"15min":2.548},"min":{"1min":0.20599999999999999,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":11.756,"5min":15.964,"15min":18.321000000000002},"last":0.17899999999999999},{"interface":"front","average":{"1min":2.0339999999999998,"5min":1.7729999999999999,"15min":2.157},"min":{"1min":0.219,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":11.907999999999999,"5min":11.907999999999999,"15min":30.128},"last":0.20499999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":2.1200000000000001,"15min":2.4049999999999998},"min":{"1min":0.29399999999999998,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.856,"5min":12.064,"15min":18.768999999999998},"last":2.8580000000000001},{"interface":"front","average":{"1min":1.6659999999999999,"5min":2.2189999999999999,"15min":2.3180000000000001},"min":{"1min":0.245,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":10.58,"5min":11.821,"15min":18.751999999999999},"last":2.891}]}]},{"osd":0,"up_from":8,"seq":34359738988,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4542016,"kb_used_data":3077496,"kb_used_omap":2116,"kb_used_meta":1462395,"kb_avail":89829824,"statfs":{"total":96636764160,"available":91985739776,"internally_reserved":0,"allocated":3151355904,"data_stored":34563332128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1497492991},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,5,0,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":2.4529999999999998,"5min":2.3849999999999998,"15min":3.052},"min":{"1min":0.184,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":8.8450000000000006,"5min":19.969999999999999,"15min":31.474},"last":0.23699999999999999},{"interface":"front","average":{"1min":2.4239999999999999,"5min":2.839,"15min":3.0419999999999998},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":8.9760000000000009,"5min":10.037000000000001,"15min":30.273},"last":0.26500000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:23:40 2023","interfaces":[{"interface":"back","average":{"1min":1.958,"5min":1.893,"15min":1.9410000000000001},"min":{"1min":0.23799999999999999,"5min":0.23200000000000001,"15min":0.182},"max":{"1min":8.0709999999999997,"5min":13.058,"15min":13.209},"last":1.0760000000000001},{"interface":"front","average":{"1min":2.2730000000000001,"5min":2.4710000000000001,"15min":2.1629999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":8.2050000000000001,"5min":13.459,"15min":16.033999999999999},"last":0.35699999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738964,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7248184,"kb_used_data":4995184,"kb_used_omap":3636,"kb_used_meta":2249355,"kb_avail":87123656,"statfs":{"total":96636764160,"available":89214623744,"internally_reserved":0,"allocated":5115068416,"data_stored":53942689996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":2303339627},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,4,5,15,7,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.9609999999999999,"5min":3.7869999999999999,"15min":3.0680000000000001},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":17.178999999999998,"5min":30.896999999999998,"15min":30.896999999999998},"last":17.178999999999998},{"interface":"front","average":{"1min":3.056,"5min":2.8620000000000001,"15min":2.661},"min":{"1min":0.27500000000000002,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":24.917000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":1.0740000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":2.254,"15min":2.1070000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":7.7910000000000004,"5min":13.358000000000001,"15min":21.326000000000001},"last":1.175},{"interface":"front","average":{"1min":3.3599999999999999,"5min":2.694,"15min":2.7189999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":24.867999999999999,"5min":24.867999999999999,"15min":24.867999999999999},"last":1.161}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3147440128,"data_stored":34567269477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5111308288,"data_stored":53948601617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1962557440,"data_stored":19343591596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:24:35.094 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:35.095 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:35.095 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:35.095 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:35.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:24:35.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.105+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:35.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.105+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:35.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.105+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:35.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.105+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:35.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.177+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:35.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.177+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:35.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.185+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:35.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.205+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:35.258 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:35.254+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:35.764 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:24:35.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:24:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:24:36.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:24:36.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.670+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:36.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.670+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:36.737 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.727+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:36.737 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.727+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:36.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.769+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:36.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.797+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:36.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:36.903+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:37.274 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:24:37.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:24:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:24:37.983 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:24:39.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.781+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:39.814 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.810+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:39.814 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.810+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:39.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.861+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:39.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.861+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:39.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.866+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:39.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.866+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:39.872 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.866+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:39.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.959+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:39.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:39.993+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:41.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.463+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:41.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.496+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:41.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.507+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:41.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.522+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:41.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.563+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:41.565 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.563+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:41.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.575+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:41.611 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.606+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:41.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:41.706+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:42.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.797+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:42.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.797+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:42.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.797+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:42.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.797+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:42.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.816+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:42.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:42.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.850+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:42.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.850+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:42.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.868+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:42.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.912+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:42.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:42.912+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:44.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.005+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:44.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.005+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:44.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.026+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:44.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.048+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:44.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.048+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:44.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.048+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:44.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.048+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:44.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.073+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:44.076 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:44.073+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:46.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.199+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:46.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.254+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:46.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.417+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:46.430 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.422+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:46.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.443+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:46.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.443+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:46.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.443+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:46.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.443+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:46.453 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.452+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:46.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.738+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:46.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.738+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:46.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.740+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:46.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.740+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:46.768 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:46.767+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:24:46.768 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:46.767+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:24:46.768 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:46.767+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:24:46.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:46.879+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.104+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.104+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.104+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.104+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.104+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:49.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.105+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:49.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.105+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:49.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.105+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:49.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.105+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:49.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.380+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:49.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.380+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:49.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.428+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:24:49.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.428+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:49.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.428+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:49.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.429+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:49.544 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.542+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:49.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.548+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:49.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.554+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:49.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.554+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:49.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.564+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:49.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.614+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:49.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:49.643+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:52.123 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:52.121+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:24:52.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.751+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:52.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.751+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:52.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:52.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.786+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:52.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.812+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:52.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.843+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:52.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:52.872+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:53.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.770+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:53.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.770+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:53.811 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:53.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:53.804+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:24:55.887 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:55.883+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:24:55.925 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:55.923+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:24:55.947 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:24:55.945+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:24:57.984 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:24:58.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:58.465+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:24:58.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:58.465+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:24:58.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:58.465+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:24:58.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:58.465+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:24:58.467 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:24:58.465+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:24:59.046 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:24:59.054 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:24:59.074 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1895,"stamp":"2023-10-03T18:24:57.453375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23636940593,"num_objects":62591,"num_object_clones":57325,"num_object_copies":125182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62591,"num_whiteouts":0,"num_read":1382341,"num_read_kb":5026450,"num_write":1259521,"num_write_kb":5023347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80517,"ondisk_log_size":80517,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14797748,"kb_used_data":10047000,"kb_used_omap":7129,"kb_used_meta":4743590,"kb_avail":268317772,"statfs":{"total":289910292480,"available":274757398528,"internally_reserved":0,"allocated":10288128000,"data_stored":107916312147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7300370,"internal_metadata":4857436910},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,5,6,15,6,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8941568,"num_objects":-51,"num_object_clones":-51,"num_object_copies":-102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-51,"num_whiteouts":0,"num_read":2129,"num_read_kb":8492,"num_write":2043,"num_write_kb":8172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.169994"},"pg_stats":[{"pgid":"2.7","version":"337'187219","reported_seq":332949,"reported_epoch":337,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:53.195821+0000","last_change":"2023-10-03T18:24:51.475400+0000","last_active":"2023-10-03T18:24:53.195821+0000","last_peered":"2023-10-03T18:24:53.195821+0000","last_clean":"2023-10-03T18:24:53.195821+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:24:53.195821+0000","last_undegraded":"2023-10-03T18:24:53.195821+0000","last_fullsized":"2023-10-03T18:24:53.195821+0000","mapping_epoch":12,"log_start":"292'177137","ondisk_log_start":"292'177137","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":259,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:15:24.143759+0000","scrub_duration":505.38444223300002,"objects_trimmed":10,"snaptrim_duration":1.717807605,"stat_sum":{"num_bytes":2943865266,"num_objects":11013,"num_object_clones":10363,"num_object_copies":22026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11013,"num_whiteouts":0,"num_read":168676,"num_read_kb":625291,"num_write":156344,"num_write_kb":623698,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15e","length":"1"}]},{"pgid":"2.6","version":"337'173540","reported_seq":308930,"reported_epoch":337,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:53.096915+0000","last_change":"2023-10-03T18:24:51.475305+0000","last_active":"2023-10-03T18:24:53.096915+0000","last_peered":"2023-10-03T18:24:53.096915+0000","last_clean":"2023-10-03T18:24:53.096915+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:24:53.096915+0000","last_undegraded":"2023-10-03T18:24:53.096915+0000","last_fullsized":"2023-10-03T18:24:53.096915+0000","mapping_epoch":12,"log_start":"289'163502","ondisk_log_start":"289'163502","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":174,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.0000881059999998,"stat_sum":{"num_bytes":2720539196,"num_objects":7793,"num_object_clones":7190,"num_object_copies":15586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7793,"num_whiteouts":0,"num_read":156749,"num_read_kb":577534,"num_write":144604,"num_write_kb":577272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"6"}]},{"pgid":"2.5","version":"337'208296","reported_seq":367781,"reported_epoch":337,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:53.107855+0000","last_change":"2023-10-03T18:24:50.665791+0000","last_active":"2023-10-03T18:24:53.107855+0000","last_peered":"2023-10-03T18:24:53.107855+0000","last_clean":"2023-10-03T18:24:53.107855+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:24:53.107855+0000","last_undegraded":"2023-10-03T18:24:53.107855+0000","last_fullsized":"2023-10-03T18:24:53.107855+0000","mapping_epoch":12,"log_start":"302'198198","ondisk_log_start":"302'198198","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":203,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":8,"snaptrim_duration":0.075128559999999997,"stat_sum":{"num_bytes":3190866524,"num_objects":10092,"num_object_clones":9383,"num_object_copies":20184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10092,"num_whiteouts":0,"num_read":192274,"num_read_kb":682695,"num_write":170406,"num_write_kb":678657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"}]},{"pgid":"2.0","version":"337'200164","reported_seq":351493,"reported_epoch":337,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:24:53.058306+0000","last_change":"2023-10-03T18:24:51.135593+0000","last_active":"2023-10-03T18:24:53.058306+0000","last_peered":"2023-10-03T18:24:53.058306+0000","last_clean":"2023-10-03T18:24:53.058306+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:24:53.058306+0000","last_undegraded":"2023-10-03T18:24:53.058306+0000","last_fullsized":"2023-10-03T18:24:53.058306+0000","mapping_epoch":12,"log_start":"305'190122","ondisk_log_start":"305'190122","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":205,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:30:12.536337+0000","scrub_duration":734.24538164700004,"objects_trimmed":10,"snaptrim_duration":3.6764982349999999,"stat_sum":{"num_bytes":3082760999,"num_objects":9826,"num_object_clones":9138,"num_object_copies":19652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9826,"num_whiteouts":0,"num_read":175516,"num_read_kb":651725,"num_write":164415,"num_write_kb":656021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"3"}]},{"pgid":"1.0","version":"88'99","reported_seq":500,"reported_epoch":336,"state":"active+clean","last_fresh":"2023-10-03T18:24:47.493124+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:24:47.493124+0000","last_peered":"2023-10-03T18:24:47.493124+0000","last_clean":"2023-10-03T18:24:47.493124+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:24:47.493124+0000","last_undegraded":"2023-10-03T18:24:47.493124+0000","last_fullsized":"2023-10-03T18:24:47.493124+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"337'200617","reported_seq":353938,"reported_epoch":337,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:53.091094+0000","last_change":"2023-10-03T18:24:50.665746+0000","last_active":"2023-10-03T18:24:53.091094+0000","last_peered":"2023-10-03T18:24:53.091094+0000","last_clean":"2023-10-03T18:24:53.091094+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:24:53.091094+0000","last_undegraded":"2023-10-03T18:24:53.091094+0000","last_fullsized":"2023-10-03T18:24:53.091094+0000","mapping_epoch":12,"log_start":"300'190608","ondisk_log_start":"300'190608","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":193,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.1902135820000002,"stat_sum":{"num_bytes":3135660874,"num_objects":9593,"num_object_clones":8894,"num_object_copies":19186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9593,"num_whiteouts":0,"num_read":178748,"num_read_kb":663184,"num_write":166824,"num_write_kb":665572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"7"}]},{"pgid":"2.1","version":"337'195745","reported_seq":335119,"reported_epoch":337,"state":"active+clean","last_fresh":"2023-10-03T18:24:53.057644+0000","last_change":"2023-10-03T18:24:40.585412+0000","last_active":"2023-10-03T18:24:53.057644+0000","last_peered":"2023-10-03T18:24:53.057644+0000","last_clean":"2023-10-03T18:24:53.057644+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:24:53.057644+0000","last_undegraded":"2023-10-03T18:24:53.057644+0000","last_fullsized":"2023-10-03T18:24:53.057644+0000","mapping_epoch":12,"log_start":"303'185685","ondisk_log_start":"303'185685","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10060,"log_dups_size":0,"ondisk_log_size":10060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:47:16.974230+0000","scrub_duration":424.08306342600002,"objects_trimmed":58,"snaptrim_duration":3.180649984,"stat_sum":{"num_bytes":2856354363,"num_objects":2971,"num_object_clones":2337,"num_object_copies":5942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2971,"num_whiteouts":0,"num_read":168218,"num_read_kb":606340,"num_write":152062,"num_write_kb":606006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"}]},{"pgid":"2.2","version":"337'196054","reported_seq":340292,"reported_epoch":337,"state":"active+clean","last_fresh":"2023-10-03T18:24:54.422655+0000","last_change":"2023-10-03T18:24:41.977303+0000","last_active":"2023-10-03T18:24:54.422655+0000","last_peered":"2023-10-03T18:24:54.422655+0000","last_clean":"2023-10-03T18:24:54.422655+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:24:54.422655+0000","last_undegraded":"2023-10-03T18:24:54.422655+0000","last_fullsized":"2023-10-03T18:24:54.422655+0000","mapping_epoch":12,"log_start":"303'186043","ondisk_log_start":"303'186043","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T06:02:56.737492+0000","scrub_duration":370.574046178,"objects_trimmed":67,"snaptrim_duration":4.5516261570000003,"stat_sum":{"num_bytes":2860253957,"num_objects":2986,"num_object_clones":2344,"num_object_copies":5972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":177232,"num_read_kb":616942,"num_write":153453,"num_write_kb":610839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"}]},{"pgid":"2.4","version":"337'183171","reported_seq":324445,"reported_epoch":337,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:24:53.106385+0000","last_change":"2023-10-03T18:24:49.855368+0000","last_active":"2023-10-03T18:24:53.106385+0000","last_peered":"2023-10-03T18:24:53.106385+0000","last_clean":"2023-10-03T18:24:53.106385+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:24:53.106385+0000","last_undegraded":"2023-10-03T18:24:53.106385+0000","last_fullsized":"2023-10-03T18:24:53.106385+0000","mapping_epoch":12,"log_start":"291'173093","ondisk_log_start":"291'173093","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":180,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.3770117810000002,"stat_sum":{"num_bytes":2846049046,"num_objects":8315,"num_object_clones":7676,"num_object_copies":16630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8315,"num_whiteouts":0,"num_read":164822,"num_read_kb":602648,"num_write":151276,"num_write_kb":603388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"15e","length":"9"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23636350225,"num_objects":62589,"num_object_clones":57325,"num_object_copies":125178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62589,"num_whiteouts":0,"num_read":1382235,"num_read_kb":5026359,"num_write":1259384,"num_write_kb":5021453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10278191104,"data_stored":107889367586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":76856,"internal_metadata":0},"log_size":80418,"ondisk_log_size":80418,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738994,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2757544,"kb_used_data":1929696,"kb_used_omap":1375,"kb_used_meta":826464,"kb_avail":91614296,"statfs":{"total":96636764160,"available":93813039104,"internally_reserved":0,"allocated":1976008704,"data_stored":19259791095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":846299268},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.375,"5min":2.3650000000000002,"15min":2.548},"min":{"1min":0.20599999999999999,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":11.756,"5min":15.964,"15min":18.321000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":2.0339999999999998,"5min":1.7729999999999999,"15min":2.157},"min":{"1min":0.219,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":11.907999999999999,"5min":11.907999999999999,"15min":30.128},"last":18.309999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":2.1200000000000001,"15min":2.4049999999999998},"min":{"1min":0.29399999999999998,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.856,"5min":12.064,"15min":18.768999999999998},"last":0.46400000000000002},{"interface":"front","average":{"1min":1.6659999999999999,"5min":2.2189999999999999,"15min":2.3180000000000001},"min":{"1min":0.245,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":10.58,"5min":11.821,"15min":18.751999999999999},"last":0.30399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738993,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4627372,"kb_used_data":3096288,"kb_used_omap":2116,"kb_used_meta":1528955,"kb_avail":89744468,"statfs":{"total":96636764160,"available":91898335232,"internally_reserved":0,"allocated":3170598912,"data_stored":34692600288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1565650431},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,2,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":3.5590000000000002,"5min":2.4500000000000002,"15min":3.1619999999999999},"min":{"1min":0.218,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":12.773,"5min":12.895,"15min":31.474},"last":3.242},{"interface":"front","average":{"1min":2.3279999999999998,"5min":2.6949999999999998,"15min":3.0979999999999999},"min":{"1min":0.222,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":6.9400000000000004,"5min":10.037000000000001,"15min":30.273},"last":0.24299999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.5940000000000001,"15min":1.8660000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.182},"max":{"1min":5.4370000000000003,"5min":12.752000000000001,"15min":13.209},"last":5.7699999999999996},{"interface":"front","average":{"1min":1.9530000000000001,"5min":2.2749999999999999,"15min":2.1800000000000002},"min":{"1min":0.28399999999999997,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":6.149,"5min":13.459,"15min":16.033999999999999},"last":4.3029999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738968,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7412832,"kb_used_data":5021016,"kb_used_omap":3636,"kb_used_meta":2388171,"kb_avail":86959008,"statfs":{"total":96636764160,"available":89046024192,"internally_reserved":0,"allocated":5141520384,"data_stored":53963920764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3724181,"internal_metadata":2445487211},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,5,2,10,3,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.9609999999999999,"5min":3.7869999999999999,"15min":3.0680000000000001},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":17.178999999999998,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.223},{"interface":"front","average":{"1min":3.056,"5min":2.8620000000000001,"15min":2.661},"min":{"1min":0.27500000000000002,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":24.917000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":3.7799999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":2.254,"15min":2.1070000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":7.7910000000000004,"5min":13.358000000000001,"15min":21.326000000000001},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.3599999999999999,"5min":2.694,"15min":2.7189999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":24.867999999999999,"5min":24.867999999999999,"15min":24.867999999999999},"last":4.452}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3166916608,"data_stored":34689154149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5138534400,"data_stored":53948036369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":40391,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1972740096,"data_stored":19252177068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:24:59.076 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:59.076 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:59.077 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:59.077 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:24:59.077 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:25:00.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.447+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:00.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.447+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:00.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.447+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:00.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.582+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:00.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.660+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:00.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.668+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:00.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.668+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:00.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.692+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:00.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.744+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:00.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:00.745+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:02.536 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.534+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:02.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.548+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:02.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.580+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:02.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.676+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:02.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.676+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:02.763 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.762+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:02.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.762+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:02.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.838+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:02.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.847+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:25:02.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.872+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:02.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.872+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:02.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.872+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:02.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.873+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:02.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.932+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:02.938 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.932+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:02.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:02.970+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:03.093 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:03.090+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:03.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:03.117+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:12.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:12.585+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:12.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:12.596+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:12.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:12.642+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:12.662 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:12.659+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:13.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.066+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:13.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.138+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:25:13.139 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.139+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:13.165 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.163+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:13.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.170+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:13.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.182+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:13.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.634+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:13.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.665+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:13.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.665+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:13.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.666+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:13.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.666+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:13.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:13.666+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:14.057 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.053+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:14.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.130+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:14.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.130+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:14.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.130+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:14.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.358+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:14.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.377+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:14.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.377+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:14.468 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.468+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:14.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.468+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:14.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:14.507+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:17.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.379+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:17.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.379+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:17.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.385+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:17.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.385+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:17.449 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.449+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:17.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.488+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:17.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:17.694+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:18.141 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.140+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:18.142 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.140+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:18.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.210+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:18.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.265+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:18.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.406+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:18.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.406+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:18.424 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.419+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:18.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.492+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:18.537 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:18.536+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:19.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:25:19.894 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:25:19.897 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:25:19.915 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1915,"stamp":"2023-10-03T18:25:19.302109+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23663310641,"num_objects":63741,"num_object_clones":58475,"num_object_copies":127482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63741,"num_whiteouts":0,"num_read":1391834,"num_read_kb":5054730,"num_write":1266035,"num_write_kb":5048683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80402,"ondisk_log_size":80402,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14551444,"kb_used_data":10097720,"kb_used_omap":7395,"kb_used_meta":4446300,"kb_avail":268564076,"statfs":{"total":289910292480,"available":275009613824,"internally_reserved":0,"allocated":10340065280,"data_stored":110435345581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7573456,"internal_metadata":4553011248},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,7,5,30,7,15,6],"upper_bound":1024},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10460980,"num_objects":1148,"num_object_clones":1152,"num_object_copies":2296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1148,"num_whiteouts":0,"num_read":2658,"num_read_kb":9948,"num_write":2506,"num_write_kb":9890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003421"},"pg_stats":[{"pgid":"2.7","version":"347'188618","reported_seq":334918,"reported_epoch":347,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:16.214898+0000","last_change":"2023-10-03T18:25:14.712680+0000","last_active":"2023-10-03T18:25:16.214898+0000","last_peered":"2023-10-03T18:25:16.214898+0000","last_clean":"2023-10-03T18:25:16.214898+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:25:16.214898+0000","last_undegraded":"2023-10-03T18:25:16.214898+0000","last_fullsized":"2023-10-03T18:25:16.214898+0000","mapping_epoch":12,"log_start":"303'178540","ondisk_log_start":"303'178540","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":275,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:08.762912+0000","scrub_duration":505.38444223300002,"objects_trimmed":22,"snaptrim_duration":2.1862656440000001,"stat_sum":{"num_bytes":2947297765,"num_objects":11208,"num_object_clones":10557,"num_object_copies":22416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11208,"num_whiteouts":0,"num_read":169778,"num_read_kb":628786,"num_write":157193,"num_write_kb":627026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"162","length":"1"}]},{"pgid":"2.6","version":"347'174747","reported_seq":310709,"reported_epoch":347,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:16.250188+0000","last_change":"2023-10-03T18:25:15.906607+0000","last_active":"2023-10-03T18:25:16.250188+0000","last_peered":"2023-10-03T18:25:16.250188+0000","last_clean":"2023-10-03T18:25:16.250188+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:25:16.250188+0000","last_undegraded":"2023-10-03T18:25:16.250188+0000","last_fullsized":"2023-10-03T18:25:16.250188+0000","mapping_epoch":12,"log_start":"297'164707","ondisk_log_start":"297'164707","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":190,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":19,"snaptrim_duration":3.4089325260000001,"stat_sum":{"num_bytes":2723476079,"num_objects":7971,"num_object_clones":7367,"num_object_copies":15942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7971,"num_whiteouts":0,"num_read":157720,"num_read_kb":580706,"num_write":145324,"num_write_kb":580113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"}]},{"pgid":"2.5","version":"347'209734","reported_seq":369893,"reported_epoch":347,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:16.118925+0000","last_change":"2023-10-03T18:25:12.642035+0000","last_active":"2023-10-03T18:25:16.118925+0000","last_peered":"2023-10-03T18:25:16.118925+0000","last_clean":"2023-10-03T18:25:16.118925+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:25:16.118925+0000","last_undegraded":"2023-10-03T18:25:16.118925+0000","last_fullsized":"2023-10-03T18:25:16.118925+0000","mapping_epoch":12,"log_start":"305'199706","ondisk_log_start":"305'199706","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":219,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.17105620299999999,"stat_sum":{"num_bytes":3194388982,"num_objects":10297,"num_object_clones":9590,"num_object_copies":20594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10297,"num_whiteouts":0,"num_read":193695,"num_read_kb":686542,"num_write":171293,"num_write_kb":682074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"1"}]},{"pgid":"2.0","version":"347'202122","reported_seq":353984,"reported_epoch":347,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:18.199045+0000","last_change":"2023-10-03T18:25:17.526978+0000","last_active":"2023-10-03T18:25:18.199045+0000","last_peered":"2023-10-03T18:25:18.199045+0000","last_clean":"2023-10-03T18:25:18.199045+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:25:18.199045+0000","last_undegraded":"2023-10-03T18:25:18.199045+0000","last_fullsized":"2023-10-03T18:25:18.199045+0000","mapping_epoch":12,"log_start":"315'192030","ondisk_log_start":"315'192030","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10092,"log_dups_size":0,"ondisk_log_size":10092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":213,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:49:48.090173+0000","scrub_duration":734.24538164700004,"objects_trimmed":78,"snaptrim_duration":5.0569033259999996,"stat_sum":{"num_bytes":3086377767,"num_objects":9799,"num_object_clones":9111,"num_object_copies":19598,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9799,"num_whiteouts":0,"num_read":176702,"num_read_kb":655535,"num_write":165310,"num_write_kb":659515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"7"}]},{"pgid":"1.0","version":"88'99","reported_seq":510,"reported_epoch":346,"state":"active+clean","last_fresh":"2023-10-03T18:25:07.532318+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:25:07.532318+0000","last_peered":"2023-10-03T18:25:07.532318+0000","last_clean":"2023-10-03T18:25:07.532318+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:25:07.532318+0000","last_undegraded":"2023-10-03T18:25:07.532318+0000","last_fullsized":"2023-10-03T18:25:07.532318+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"347'201948","reported_seq":355898,"reported_epoch":347,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:15.906744+0000","last_change":"2023-10-03T18:25:15.906744+0000","last_active":"2023-10-03T18:25:15.906744+0000","last_peered":"2023-10-03T18:25:15.906744+0000","last_clean":"2023-10-03T18:25:15.906744+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:25:15.906744+0000","last_undegraded":"2023-10-03T18:25:15.906744+0000","last_fullsized":"2023-10-03T18:25:15.906744+0000","mapping_epoch":12,"log_start":"304'191912","ondisk_log_start":"304'191912","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":210,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":2.8229514880000002,"stat_sum":{"num_bytes":3139097468,"num_objects":9815,"num_object_clones":9115,"num_object_copies":19630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9815,"num_whiteouts":0,"num_read":179913,"num_read_kb":666917,"num_write":167645,"num_write_kb":668774,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"}]},{"pgid":"2.1","version":"347'197418","reported_seq":337302,"reported_epoch":347,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:18.165083+0000","last_change":"2023-10-03T18:25:16.741073+0000","last_active":"2023-10-03T18:25:18.165083+0000","last_peered":"2023-10-03T18:25:18.165083+0000","last_clean":"2023-10-03T18:25:18.165083+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:25:18.165083+0000","last_undegraded":"2023-10-03T18:25:18.165083+0000","last_fullsized":"2023-10-03T18:25:18.165083+0000","mapping_epoch":12,"log_start":"305'187397","ondisk_log_start":"305'187397","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:33:01.072754+0000","scrub_duration":424.08306342600002,"objects_trimmed":68,"snaptrim_duration":4.2830953669999996,"stat_sum":{"num_bytes":2859872777,"num_objects":3092,"num_object_clones":2459,"num_object_copies":6184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3092,"num_whiteouts":0,"num_read":169452,"num_read_kb":609907,"num_write":152940,"num_write_kb":609420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"7"}]},{"pgid":"2.2","version":"347'197353","reported_seq":342197,"reported_epoch":347,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:14.703663+0000","last_change":"2023-10-03T18:25:14.220955+0000","last_active":"2023-10-03T18:25:14.703663+0000","last_peered":"2023-10-03T18:25:14.703663+0000","last_clean":"2023-10-03T18:25:14.703663+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:25:14.703663+0000","last_undegraded":"2023-10-03T18:25:14.703663+0000","last_fullsized":"2023-10-03T18:25:14.703663+0000","mapping_epoch":12,"log_start":"305'187352","ondisk_log_start":"305'187352","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:43:55.003144+0000","scrub_duration":370.574046178,"objects_trimmed":32,"snaptrim_duration":1.732250624,"stat_sum":{"num_bytes":2862920605,"num_objects":3074,"num_object_clones":2429,"num_object_copies":6148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3074,"num_whiteouts":0,"num_read":178561,"num_read_kb":620308,"num_write":154142,"num_write_kb":613456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"6"}]},{"pgid":"2.4","version":"347'184508","reported_seq":326335,"reported_epoch":347,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:16.122796+0000","last_change":"2023-10-03T18:25:12.642290+0000","last_active":"2023-10-03T18:25:16.122796+0000","last_peered":"2023-10-03T18:25:16.122796+0000","last_clean":"2023-10-03T18:25:16.122796+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:25:16.122796+0000","last_undegraded":"2023-10-03T18:25:16.122796+0000","last_fullsized":"2023-10-03T18:25:16.122796+0000","mapping_epoch":12,"log_start":"303'174501","ondisk_log_start":"303'174501","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10007,"log_dups_size":0,"ondisk_log_size":10007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":197,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":32,"snaptrim_duration":0.78783904999999999,"stat_sum":{"num_bytes":2849288830,"num_objects":8483,"num_object_clones":7847,"num_object_copies":16966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8483,"num_whiteouts":0,"num_read":165907,"num_read_kb":605938,"num_write":152051,"num_write_kb":606411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"162","length":"5"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23662720273,"num_objects":63739,"num_object_clones":58475,"num_object_copies":127478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63739,"num_whiteouts":0,"num_read":1391728,"num_read_kb":5054639,"num_write":1265898,"num_write_kb":5046789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10328207360,"data_stored":110511827490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80770,"internal_metadata":0},"log_size":80303,"ondisk_log_size":80303,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738999,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2818320,"kb_used_data":1940552,"kb_used_omap":1375,"kb_used_meta":876384,"kb_avail":91553520,"statfs":{"total":96636764160,"available":93750804480,"internally_reserved":0,"allocated":1987125248,"data_stored":19676113605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":897417348},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.375,"5min":2.3650000000000002,"15min":2.548},"min":{"1min":0.20599999999999999,"5min":0.17000000000000001,"15min":0.16600000000000001},"max":{"1min":11.756,"5min":15.964,"15min":18.321000000000002},"last":12.737},{"interface":"front","average":{"1min":2.0339999999999998,"5min":1.7729999999999999,"15min":2.157},"min":{"1min":0.219,"5min":0.21299999999999999,"15min":0.20100000000000001},"max":{"1min":11.907999999999999,"5min":11.907999999999999,"15min":30.128},"last":5.8819999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:24:19 2023","interfaces":[{"interface":"back","average":{"1min":2.3410000000000002,"5min":2.1200000000000001,"15min":2.4049999999999998},"min":{"1min":0.29399999999999998,"5min":0.221,"15min":0.21199999999999999},"max":{"1min":11.856,"5min":12.064,"15min":18.768999999999998},"last":12.368},{"interface":"front","average":{"1min":1.6659999999999999,"5min":2.2189999999999999,"15min":2.3180000000000001},"min":{"1min":0.245,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":10.58,"5min":11.821,"15min":18.751999999999999},"last":12.449}]}]},{"osd":0,"up_from":8,"seq":34359738997,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4727148,"kb_used_data":3110052,"kb_used_omap":2116,"kb_used_meta":1614971,"kb_avail":89644692,"statfs":{"total":96636764160,"available":91796164608,"internally_reserved":0,"allocated":3184693248,"data_stored":35509428382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1653730815},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,0,4,1,5,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":3.5590000000000002,"5min":2.4500000000000002,"15min":3.1619999999999999},"min":{"1min":0.218,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":12.773,"5min":12.895,"15min":31.474},"last":1.3340000000000001},{"interface":"front","average":{"1min":2.3279999999999998,"5min":2.6949999999999998,"15min":3.0979999999999999},"min":{"1min":0.222,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":6.9400000000000004,"5min":10.037000000000001,"15min":30.273},"last":2.8210000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.5940000000000001,"15min":1.8660000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.182},"max":{"1min":5.4370000000000003,"5min":12.752000000000001,"15min":13.209},"last":8.9220000000000006},{"interface":"front","average":{"1min":1.9530000000000001,"5min":2.2749999999999999,"15min":2.1800000000000002},"min":{"1min":0.28399999999999997,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":6.149,"5min":13.459,"15min":16.033999999999999},"last":2.8820000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738972,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7005976,"kb_used_data":5047116,"kb_used_omap":3903,"kb_used_meta":1954944,"kb_avail":87365864,"statfs":{"total":96636764160,"available":89462644736,"internally_reserved":0,"allocated":5168246784,"data_stored":55249803594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3997267,"internal_metadata":2001863085},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,5,4,26,6,10,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.9609999999999999,"5min":3.7869999999999999,"15min":3.0680000000000001},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":17.178999999999998,"5min":30.896999999999998,"15min":30.896999999999998},"last":4.6340000000000003},{"interface":"front","average":{"1min":3.056,"5min":2.8620000000000001,"15min":2.661},"min":{"1min":0.27500000000000002,"5min":0.221,"15min":0.17799999999999999},"max":{"1min":24.917000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":4.6680000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:24:30 2023","interfaces":[{"interface":"back","average":{"1min":2.6760000000000002,"5min":2.254,"15min":2.1070000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.19400000000000001},"max":{"1min":7.7910000000000004,"5min":13.358000000000001,"15min":21.326000000000001},"last":0.315},{"interface":"front","average":{"1min":3.3599999999999999,"5min":2.694,"15min":2.7189999999999999},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":24.867999999999999,"5min":24.867999999999999,"15min":24.867999999999999},"last":0.43099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3180711936,"data_stored":35549084773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5164150784,"data_stored":55282836753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44305,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1983344640,"data_stored":19679905964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:25:19.917 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:19.917 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:19.917 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:19.917 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:19.917 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:25:22.252 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.251+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:22.289 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.289+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:22.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.290+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:22.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.290+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:22.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.290+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:22.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.291+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:22.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:22.291+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:25.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.296+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:25.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.392+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:25.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.399+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:25.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.543+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:25.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.543+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:25.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.564+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:25.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.564+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:25.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:25.607+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:28.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.554+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:28.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.592+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:28.685 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.684+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:28.751 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.750+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:28.754 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.753+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:28.754 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.753+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:28.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.906+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:28.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.921+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:28.929 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:28.922+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:29.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.495+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:29.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.583+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:29.586 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.583+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:29.599 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.599+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:29.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.603+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:29.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.603+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:29.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.669+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:29.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.673+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:29.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:29.724+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:30.791 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.789+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:25:30.846 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.845+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:30.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.847+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:30.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.847+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:30.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.853+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:30.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:30.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:30.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:30.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:30.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:30.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.932+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:30.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.933+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:30.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.933+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:30.938 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.933+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:30.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.952+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:30.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.963+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:30.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:30.963+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:31.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:31.017+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:31.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:31.021+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:31.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:31.021+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:33.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.000+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:25:33.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.016+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:33.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:33.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.092+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:33.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.092+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:33.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.173+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:33.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.185+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:33.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.185+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:33.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.185+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:33.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.185+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:33.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:33.185+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:35.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.522+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:35.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.559+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:35.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.559+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:35.814 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.813+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:35.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.836+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:35.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.876+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:35.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.876+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:35.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.923+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:35.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.923+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:35.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:35.996+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:38.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.509+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:38.585 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.584+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:38.672 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:38.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.707+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:38.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.707+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:38.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.742+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:38.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.743+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:38.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.743+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:38.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.743+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:38.842 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:38.841+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:39.919 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:25:40.657 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:25:40.676 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:25:40.696 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1928,"stamp":"2023-10-03T18:25:39.314694+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23686788913,"num_objects":64274,"num_object_clones":59008,"num_object_copies":128548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64274,"num_whiteouts":0,"num_read":1398954,"num_read_kb":5078505,"num_write":1271675,"num_write_kb":5071243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80498,"ondisk_log_size":80498,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14938456,"kb_used_data":10140860,"kb_used_omap":7395,"kb_used_meta":4790172,"kb_avail":268177064,"statfs":{"total":289910292480,"available":274613313536,"internally_reserved":0,"allocated":10384240640,"data_stored":111873866429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7573456,"internal_metadata":4905136176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,19,7,14,16,11,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11608064,"num_objects":31,"num_object_clones":31,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":4215,"num_read_kb":12629,"num_write":2791,"num_write_kb":11164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006549"},"pg_stats":[{"pgid":"2.7","version":"350'189825","reported_seq":336619,"reported_epoch":350,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:38.289027+0000","last_change":"2023-10-03T18:25:38.175962+0000","last_active":"2023-10-03T18:25:38.289027+0000","last_peered":"2023-10-03T18:25:38.289027+0000","last_clean":"2023-10-03T18:25:38.289027+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:25:38.289027+0000","last_undegraded":"2023-10-03T18:25:38.289027+0000","last_fullsized":"2023-10-03T18:25:38.289027+0000","mapping_epoch":12,"log_start":"305'179744","ondisk_log_start":"305'179744","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":272,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:08.762912+0000","scrub_duration":505.38444223300002,"objects_trimmed":9,"snaptrim_duration":0.64711344900000001,"stat_sum":{"num_bytes":2950382053,"num_objects":11352,"num_object_clones":10701,"num_object_copies":22704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11352,"num_whiteouts":0,"num_read":170649,"num_read_kb":631842,"num_write":157939,"num_write_kb":630010,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"165","length":"1"}]},{"pgid":"2.6","version":"350'175785","reported_seq":312208,"reported_epoch":350,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:38.220668+0000","last_change":"2023-10-03T18:25:38.220718+0000","last_active":"2023-10-03T18:25:38.220668+0000","last_peered":"2023-10-03T18:25:38.220668+0000","last_clean":"2023-10-03T18:25:38.220668+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:25:38.220668+0000","last_undegraded":"2023-10-03T18:25:38.220668+0000","last_fullsized":"2023-10-03T18:25:38.220668+0000","mapping_epoch":12,"log_start":"304'165710","ondisk_log_start":"304'165710","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":187,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.093211844,"stat_sum":{"num_bytes":2726113903,"num_objects":8122,"num_object_clones":7518,"num_object_copies":16244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":158519,"num_read_kb":583431,"num_write":145964,"num_write_kb":582673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"3"}]},{"pgid":"2.5","version":"350'211048","reported_seq":371737,"reported_epoch":350,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:38.170582+0000","last_change":"2023-10-03T18:25:35.480400+0000","last_active":"2023-10-03T18:25:38.170582+0000","last_peered":"2023-10-03T18:25:38.170582+0000","last_clean":"2023-10-03T18:25:38.170582+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:25:38.170582+0000","last_undegraded":"2023-10-03T18:25:38.170582+0000","last_fullsized":"2023-10-03T18:25:38.170582+0000","mapping_epoch":12,"log_start":"307'201010","ondisk_log_start":"307'201010","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":216,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.9630286959999999,"stat_sum":{"num_bytes":3197678070,"num_objects":10452,"num_object_clones":9745,"num_object_copies":20904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10452,"num_whiteouts":0,"num_read":194733,"num_read_kb":689830,"num_write":172076,"num_write_kb":685206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"4"}]},{"pgid":"2.0","version":"350'203640","reported_seq":355935,"reported_epoch":350,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:38.480288+0000","last_change":"2023-10-03T18:25:37.775982+0000","last_active":"2023-10-03T18:25:38.480288+0000","last_peered":"2023-10-03T18:25:38.480288+0000","last_clean":"2023-10-03T18:25:38.480288+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:25:38.480288+0000","last_undegraded":"2023-10-03T18:25:38.480288+0000","last_fullsized":"2023-10-03T18:25:38.480288+0000","mapping_epoch":12,"log_start":"317'193635","ondisk_log_start":"317'193635","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":204,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:49:48.090173+0000","scrub_duration":734.24538164700004,"objects_trimmed":18,"snaptrim_duration":0.27151809999999998,"stat_sum":{"num_bytes":3089240871,"num_objects":9720,"num_object_clones":9032,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9720,"num_whiteouts":0,"num_read":177568,"num_read_kb":658567,"num_write":165994,"num_write_kb":662251,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"4"}]},{"pgid":"1.0","version":"88'99","reported_seq":513,"reported_epoch":349,"state":"active+clean","last_fresh":"2023-10-03T18:25:32.503436+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:25:32.503436+0000","last_peered":"2023-10-03T18:25:32.503436+0000","last_clean":"2023-10-03T18:25:32.503436+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:25:32.503436+0000","last_undegraded":"2023-10-03T18:25:32.503436+0000","last_fullsized":"2023-10-03T18:25:32.503436+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"350'203221","reported_seq":357674,"reported_epoch":350,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:38.170811+0000","last_change":"2023-10-03T18:25:35.823528+0000","last_active":"2023-10-03T18:25:38.170811+0000","last_peered":"2023-10-03T18:25:38.170811+0000","last_clean":"2023-10-03T18:25:38.170811+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:25:38.170811+0000","last_undegraded":"2023-10-03T18:25:38.170811+0000","last_fullsized":"2023-10-03T18:25:38.170811+0000","mapping_epoch":12,"log_start":"306'193121","ondisk_log_start":"306'193121","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":207,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.3083113279999998,"stat_sum":{"num_bytes":3142378364,"num_objects":9996,"num_object_clones":9296,"num_object_copies":19992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9996,"num_whiteouts":0,"num_read":180822,"num_read_kb":670109,"num_write":168432,"num_write_kb":671922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"1"}]},{"pgid":"2.1","version":"350'198942","reported_seq":339194,"reported_epoch":350,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:38.491682+0000","last_change":"2023-10-03T18:25:37.970650+0000","last_active":"2023-10-03T18:25:38.491682+0000","last_peered":"2023-10-03T18:25:38.491682+0000","last_clean":"2023-10-03T18:25:38.491682+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:25:38.491682+0000","last_undegraded":"2023-10-03T18:25:38.491682+0000","last_fullsized":"2023-10-03T18:25:38.491682+0000","mapping_epoch":12,"log_start":"315'188900","ondisk_log_start":"315'188900","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:33:01.072754+0000","scrub_duration":424.08306342600002,"objects_trimmed":14,"snaptrim_duration":0.46140848299999998,"stat_sum":{"num_bytes":2862604809,"num_objects":2967,"num_object_clones":2334,"num_object_copies":5934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2967,"num_whiteouts":0,"num_read":170287,"num_read_kb":612609,"num_write":153596,"num_write_kb":612044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"}]},{"pgid":"2.2","version":"349'198805","reported_seq":344038,"reported_epoch":349,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:34.885405+0000","last_change":"2023-10-03T18:25:34.277469+0000","last_active":"2023-10-03T18:25:34.885405+0000","last_peered":"2023-10-03T18:25:34.885405+0000","last_clean":"2023-10-03T18:25:34.885405+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:25:34.885405+0000","last_undegraded":"2023-10-03T18:25:34.885405+0000","last_fullsized":"2023-10-03T18:25:34.885405+0000","mapping_epoch":12,"log_start":"312'188757","ondisk_log_start":"312'188757","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:43:55.003144+0000","scrub_duration":370.574046178,"objects_trimmed":39,"snaptrim_duration":1.771801121,"stat_sum":{"num_bytes":2865460125,"num_objects":3024,"num_object_clones":2379,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":179502,"num_read_kb":623147,"num_write":154754,"num_write_kb":615904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"164","length":"3"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"4"}]},{"pgid":"2.4","version":"350'185716","reported_seq":328012,"reported_epoch":350,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:38.168229+0000","last_change":"2023-10-03T18:25:35.828345+0000","last_active":"2023-10-03T18:25:38.168229+0000","last_peered":"2023-10-03T18:25:38.168229+0000","last_clean":"2023-10-03T18:25:38.168229+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:25:38.168229+0000","last_undegraded":"2023-10-03T18:25:38.168229+0000","last_fullsized":"2023-10-03T18:25:38.168229+0000","mapping_epoch":12,"log_start":"305'175706","ondisk_log_start":"305'175706","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10010,"log_dups_size":0,"ondisk_log_size":10010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":194,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":9.0226065589999997,"stat_sum":{"num_bytes":2852340350,"num_objects":8639,"num_object_clones":8003,"num_object_copies":17278,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8639,"num_whiteouts":0,"num_read":166768,"num_read_kb":608879,"num_write":152783,"num_write_kb":609339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"165","length":"2"},{"start":"169","length":"1"},{"start":"16b","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23686198545,"num_objects":64272,"num_object_clones":59008,"num_object_copies":128544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64272,"num_whiteouts":0,"num_read":1398848,"num_read_kb":5078414,"num_write":1271538,"num_write_kb":5069349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10373644288,"data_stored":111892748834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80770,"internal_metadata":0},"log_size":80399,"ondisk_log_size":80399,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739003,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2872340,"kb_used_data":1948748,"kb_used_omap":1375,"kb_used_meta":922208,"kb_avail":91499500,"statfs":{"total":96636764160,"available":93695488000,"internally_reserved":0,"allocated":1995517952,"data_stored":19711945493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":944341124},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,2,3,4,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.665,"5min":2.4079999999999999,"15min":2.6040000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":12.737,"5min":15.964,"15min":18.321000000000002},"last":5.4329999999999998},{"interface":"front","average":{"1min":3.2109999999999999,"5min":2.141,"15min":2.258},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":18.309999999999999,"5min":18.309999999999999,"15min":30.128},"last":2.8839999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.6190000000000002,"5min":2.383,"15min":2.395},"min":{"1min":0.25,"5min":0.25,"15min":0.21199999999999999},"max":{"1min":12.368,"5min":12.368,"15min":18.768999999999998},"last":2.968},{"interface":"front","average":{"1min":3.0139999999999998,"5min":2.423,"15min":2.323},"min":{"1min":0.25,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":12.449,"5min":12.449,"15min":18.751999999999999},"last":0.254}]}]},{"osd":0,"up_from":8,"seq":34359739001,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4862620,"kb_used_data":3122640,"kb_used_omap":2116,"kb_used_meta":1737851,"kb_avail":89509220,"statfs":{"total":96636764160,"available":91657441280,"internally_reserved":0,"allocated":3197583360,"data_stored":36222876542,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1779559935},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,2,2,3,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":3.5590000000000002,"5min":2.4500000000000002,"15min":3.1619999999999999},"min":{"1min":0.218,"5min":0.17399999999999999,"15min":0.14799999999999999},"max":{"1min":12.773,"5min":12.895,"15min":31.474},"last":0.95599999999999996},{"interface":"front","average":{"1min":2.3279999999999998,"5min":2.6949999999999998,"15min":3.0979999999999999},"min":{"1min":0.222,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":6.9400000000000004,"5min":10.037000000000001,"15min":30.273},"last":2.4910000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:24:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.5940000000000001,"15min":1.8660000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.182},"max":{"1min":5.4370000000000003,"5min":12.752000000000001,"15min":13.209},"last":1.9390000000000001},{"interface":"front","average":{"1min":1.9530000000000001,"5min":2.2749999999999999,"15min":2.1800000000000002},"min":{"1min":0.28399999999999997,"5min":0.21099999999999999,"15min":0.20999999999999999},"max":{"1min":6.149,"5min":13.459,"15min":16.033999999999999},"last":2.5390000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738976,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7203496,"kb_used_data":5069472,"kb_used_omap":3903,"kb_used_meta":2130112,"kb_avail":87168344,"statfs":{"total":96636764160,"available":89260384256,"internally_reserved":0,"allocated":5191139328,"data_stored":55939044394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3997267,"internal_metadata":2181235117},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,16,3,9,9,4,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":3.1899999999999999,"5min":3.54,"15min":3.1699999999999999},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":14.814,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":3.0369999999999999,"5min":2.6829999999999998,"15min":2.77},"min":{"1min":0.255,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":13.880000000000001,"5min":24.917000000000002,"15min":24.917000000000002},"last":0.28899999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":2.3180000000000001,"15min":2.1019999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19400000000000001},"max":{"1min":11.691000000000001,"5min":13.358000000000001,"15min":21.326000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":2.177,"5min":2.5579999999999998,"15min":2.7450000000000001},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":8.0909999999999993,"5min":24.867999999999999,"15min":24.867999999999999},"last":0.247}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3193888768,"data_stored":36244466789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5187969024,"data_stored":55938499857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44305,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1991786496,"data_stored":19709782188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:25:40.698 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:40.698 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:40.698 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:40.698 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:25:40.698 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:25:43.115 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:25:43.114+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:25:43.115 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:25:43.114+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:25:43.115 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:25:43.114+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:25:43.115 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:25:43.114+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:25:44.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.093+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:44.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.128+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:44.131 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.128+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:44.306 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.305+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:44.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.313+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:44.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.316+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:44.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.369+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:44.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.369+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:44.388 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:44.387+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:45.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.378+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:45.402 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.401+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:45.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.416+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:45.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.443+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:45.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.443+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:45.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.504+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:45.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.504+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:45.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.535+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:45.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.535+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:45.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:45.535+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:46.758 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:46.752+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:25:46.758 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:46.752+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:25:48.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.550+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:48.562 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.559+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:48.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.562+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:48.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.570+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:48.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.570+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:48.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.570+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:48.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.570+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:48.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.595+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:48.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:48.595+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:51.256 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.254+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:51.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.256+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:51.269 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.268+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:51.275 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.273+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:51.294 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.292+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:51.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.310+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:51.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.310+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:51.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.371+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:51.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.379+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:51.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.424+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:51.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.490+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:51.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.574+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:51.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.679+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:51.704 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.703+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:51.765 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.751+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:51.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.769+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:51.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:51.785+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:53.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:53.688+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:25:53.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:53.688+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:25:53.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:53.688+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:25:53.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:53.688+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:25:53.691 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:25:53.688+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:25:54.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.036+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:54.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.036+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:54.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.036+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:54.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.063+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:54.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.070+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:54.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.070+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:54.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.070+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:54.072 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.070+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:54.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.146+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:54.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.146+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:54.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.686+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:54.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.827+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:54.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.844+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:54.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.844+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:54.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.844+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:54.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.844+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:54.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.847+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:54.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:54.851+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:54.926 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:25:54.924+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:25:56.700 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.692+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:56.703 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.700+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:56.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.724+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:56.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:56.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:56.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:56.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:56.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:56.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:56.737 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:56.730+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:25:59.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.793+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:25:59.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.793+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:25:59.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:25:59.797 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.793+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:25:59.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.800+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:25:59.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:25:59.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:25:59.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:25:59.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:25:59.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:25:59.813 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:25:59.801+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:00.699 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:26:00.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.754+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:00.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.898+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:00.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.914+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:00.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.948+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:00.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.948+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:00.959 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.948+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:00.994 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.992+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:01.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:00.998+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:01.485 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:26:01.494 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:26:01.511 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1944,"stamp":"2023-10-03T18:26:01.328985+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23718303537,"num_objects":63861,"num_object_clones":58595,"num_object_copies":127722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63861,"num_whiteouts":0,"num_read":1407100,"num_read_kb":5110659,"num_write":1279194,"num_write_kb":5101319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80688,"ondisk_log_size":80688,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15239364,"kb_used_data":10203100,"kb_used_omap":7395,"kb_used_meta":5028828,"kb_avail":267876156,"statfs":{"total":289910292480,"available":274305183744,"internally_reserved":0,"allocated":10447974400,"data_stored":111480505789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7573456,"internal_metadata":5149519920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,6,12,9,9,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16498688,"num_objects":-275,"num_object_clones":-275,"num_object_copies":-550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-275,"num_whiteouts":0,"num_read":4412,"num_read_kb":17286,"num_write":3922,"num_write_kb":15688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.009031"},"pg_stats":[{"pgid":"2.7","version":"354'191116","reported_seq":338820,"reported_epoch":354,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:59.768571+0000","last_change":"2023-10-03T18:25:57.244909+0000","last_active":"2023-10-03T18:25:59.768571+0000","last_peered":"2023-10-03T18:25:59.768571+0000","last_clean":"2023-10-03T18:25:59.768571+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:25:59.768571+0000","last_undegraded":"2023-10-03T18:25:59.768571+0000","last_fullsized":"2023-10-03T18:25:59.768571+0000","mapping_epoch":12,"log_start":"309'181044","ondisk_log_start":"309'181044","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":269,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:10:08.762912+0000","scrub_duration":505.38444223300002,"objects_trimmed":28,"snaptrim_duration":0.79732864999999997,"stat_sum":{"num_bytes":2954240485,"num_objects":11318,"num_object_clones":10667,"num_object_copies":22636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11318,"num_whiteouts":0,"num_read":171686,"num_read_kb":635970,"num_write":158868,"num_write_kb":633726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"354'176993","reported_seq":314142,"reported_epoch":354,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:59.768397+0000","last_change":"2023-10-03T18:25:57.240047+0000","last_active":"2023-10-03T18:25:59.768397+0000","last_peered":"2023-10-03T18:25:59.768397+0000","last_clean":"2023-10-03T18:25:59.768397+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:25:59.768397+0000","last_undegraded":"2023-10-03T18:25:59.768397+0000","last_fullsized":"2023-10-03T18:25:59.768397+0000","mapping_epoch":12,"log_start":"306'166916","ondisk_log_start":"306'166916","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":183,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.7377357680000003,"stat_sum":{"num_bytes":2729775727,"num_objects":8079,"num_object_clones":7475,"num_object_copies":16158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8079,"num_whiteouts":0,"num_read":159360,"num_read_kb":586791,"num_write":146843,"num_write_kb":586189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"7"}]},{"pgid":"2.5","version":"354'212495","reported_seq":374193,"reported_epoch":354,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:59.775831+0000","last_change":"2023-10-03T18:25:58.073807+0000","last_active":"2023-10-03T18:25:59.775831+0000","last_peered":"2023-10-03T18:25:59.775831+0000","last_clean":"2023-10-03T18:25:59.775831+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:25:59.775831+0000","last_undegraded":"2023-10-03T18:25:59.775831+0000","last_fullsized":"2023-10-03T18:25:59.775831+0000","mapping_epoch":12,"log_start":"317'202416","ondisk_log_start":"317'202416","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":212,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.53699431900000005,"stat_sum":{"num_bytes":3201970678,"num_objects":10402,"num_object_clones":9695,"num_object_copies":20804,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10402,"num_whiteouts":0,"num_read":195879,"num_read_kb":694390,"num_write":173088,"num_write_kb":689254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"8"}]},{"pgid":"2.0","version":"354'205345","reported_seq":358441,"reported_epoch":354,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:58.663184+0000","last_change":"2023-10-03T18:25:57.196872+0000","last_active":"2023-10-03T18:25:58.663184+0000","last_peered":"2023-10-03T18:25:58.663184+0000","last_clean":"2023-10-03T18:25:58.663184+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:25:58.663184+0000","last_undegraded":"2023-10-03T18:25:58.663184+0000","last_fullsized":"2023-10-03T18:25:58.663184+0000","mapping_epoch":12,"log_start":"320'195246","ondisk_log_start":"320'195246","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":194,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:49:48.090173+0000","scrub_duration":734.24538164700004,"objects_trimmed":24,"snaptrim_duration":4.6813986290000003,"stat_sum":{"num_bytes":3093152551,"num_objects":9477,"num_object_clones":8789,"num_object_copies":18954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9477,"num_whiteouts":0,"num_read":178565,"num_read_kb":662543,"num_write":166928,"num_write_kb":665987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":518,"reported_epoch":354,"state":"active+clean","last_fresh":"2023-10-03T18:25:57.563682+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:25:57.563682+0000","last_peered":"2023-10-03T18:25:57.563682+0000","last_clean":"2023-10-03T18:25:57.563682+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:25:57.563682+0000","last_undegraded":"2023-10-03T18:25:57.563682+0000","last_fullsized":"2023-10-03T18:25:57.563682+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"354'204508","reported_seq":359919,"reported_epoch":354,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:25:59.773123+0000","last_change":"2023-10-03T18:25:59.535203+0000","last_active":"2023-10-03T18:25:59.773123+0000","last_peered":"2023-10-03T18:25:59.773123+0000","last_clean":"2023-10-03T18:25:59.773123+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:25:59.773123+0000","last_undegraded":"2023-10-03T18:25:59.773123+0000","last_fullsized":"2023-10-03T18:25:59.773123+0000","mapping_epoch":12,"log_start":"315'194425","ondisk_log_start":"315'194425","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":204,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":2.827388086,"stat_sum":{"num_bytes":3146392444,"num_objects":9977,"num_object_clones":9277,"num_object_copies":19954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9977,"num_whiteouts":0,"num_read":181904,"num_read_kb":674401,"num_write":169372,"num_write_kb":675682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"4"}]},{"pgid":"2.1","version":"354'199988","reported_seq":341088,"reported_epoch":354,"state":"active+clean","last_fresh":"2023-10-03T18:25:58.603190+0000","last_change":"2023-10-03T18:25:41.274341+0000","last_active":"2023-10-03T18:25:58.603190+0000","last_peered":"2023-10-03T18:25:58.603190+0000","last_clean":"2023-10-03T18:25:58.603190+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:25:58.603190+0000","last_undegraded":"2023-10-03T18:25:58.603190+0000","last_fullsized":"2023-10-03T18:25:58.603190+0000","mapping_epoch":12,"log_start":"317'189903","ondisk_log_start":"317'189903","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:33:01.072754+0000","scrub_duration":424.08306342600002,"objects_trimmed":60,"snaptrim_duration":3.76575397,"stat_sum":{"num_bytes":2866217481,"num_objects":2998,"num_object_clones":2365,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":171205,"num_read_kb":616257,"num_write":154455,"num_write_kb":615480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"}]},{"pgid":"2.2","version":"354'200210","reported_seq":346414,"reported_epoch":354,"state":"active+clean","last_fresh":"2023-10-03T18:26:00.094683+0000","last_change":"2023-10-03T18:25:42.213574+0000","last_active":"2023-10-03T18:26:00.094683+0000","last_peered":"2023-10-03T18:26:00.094683+0000","last_clean":"2023-10-03T18:26:00.094683+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:26:00.094683+0000","last_undegraded":"2023-10-03T18:26:00.094683+0000","last_fullsized":"2023-10-03T18:26:00.094683+0000","mapping_epoch":12,"log_start":"317'190160","ondisk_log_start":"317'190160","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10050,"log_dups_size":0,"ondisk_log_size":10050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:43:55.003144+0000","scrub_duration":370.574046178,"objects_trimmed":76,"snaptrim_duration":4.7184871639999999,"stat_sum":{"num_bytes":2869711773,"num_objects":2998,"num_object_clones":2353,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":180669,"num_read_kb":627525,"num_write":155788,"num_write_kb":620040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"}]},{"pgid":"2.4","version":"354'186955","reported_seq":330121,"reported_epoch":354,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:25:59.734363+0000","last_change":"2023-10-03T18:25:59.535100+0000","last_active":"2023-10-03T18:25:59.734363+0000","last_peered":"2023-10-03T18:25:59.734363+0000","last_clean":"2023-10-03T18:25:59.734363+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:25:59.734363+0000","last_undegraded":"2023-10-03T18:25:59.734363+0000","last_fullsized":"2023-10-03T18:25:59.734363+0000","mapping_epoch":12,"log_start":"308'176911","ondisk_log_start":"308'176911","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":189,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":2.030933359,"stat_sum":{"num_bytes":2856252030,"num_objects":8610,"num_object_clones":7974,"num_object_copies":17220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8610,"num_whiteouts":0,"num_read":167726,"num_read_kb":612691,"num_write":153715,"num_write_kb":613067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23717713169,"num_objects":63859,"num_object_clones":58595,"num_object_copies":127718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63859,"num_whiteouts":0,"num_read":1406994,"num_read_kb":5110568,"num_write":1279057,"num_write_kb":5099425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10436980736,"data_stored":111435819554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80770,"internal_metadata":0},"log_size":80589,"ondisk_log_size":80589,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739007,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":2949496,"kb_used_data":1959336,"kb_used_omap":1375,"kb_used_meta":988768,"kb_avail":91422344,"statfs":{"total":96636764160,"available":93616480256,"internally_reserved":0,"allocated":2006360064,"data_stored":19560308853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1012498564},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.665,"5min":2.4079999999999999,"15min":2.6040000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":12.737,"5min":15.964,"15min":18.321000000000002},"last":10.922000000000001},{"interface":"front","average":{"1min":3.2109999999999999,"5min":2.141,"15min":2.258},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":18.309999999999999,"5min":18.309999999999999,"15min":30.128},"last":3.1139999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.6190000000000002,"5min":2.383,"15min":2.395},"min":{"1min":0.25,"5min":0.25,"15min":0.21199999999999999},"max":{"1min":12.368,"5min":12.368,"15min":18.768999999999998},"last":10.759},{"interface":"front","average":{"1min":3.0139999999999998,"5min":2.423,"15min":2.323},"min":{"1min":0.25,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":12.449,"5min":12.449,"15min":18.751999999999999},"last":10.616}]}]},{"osd":0,"up_from":8,"seq":34359739006,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4952632,"kb_used_data":3144624,"kb_used_omap":2116,"kb_used_meta":1805883,"kb_avail":89419208,"statfs":{"total":96636764160,"available":91565268992,"internally_reserved":0,"allocated":3220094976,"data_stored":36180395598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1849224703},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,4,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":5.3380000000000001,"5min":3.0470000000000002,"15min":3.226},"min":{"1min":0.25900000000000001,"5min":0.17499999999999999,"15min":0.14799999999999999},"max":{"1min":45.886000000000003,"5min":45.886000000000003,"15min":45.886000000000003},"last":7.2960000000000003},{"interface":"front","average":{"1min":5.2290000000000001,"5min":3.1429999999999998,"15min":3.2789999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.219}]},{"osd":2,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.7869999999999999,"15min":1.903},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":8.9220000000000006,"5min":12.752000000000001,"15min":13.209},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.784,"5min":2.242,"15min":2.1379999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":5.8570000000000002,"5min":13.459,"15min":16.033999999999999},"last":1.843}]}]},{"osd":1,"up_from":8,"seq":34359738980,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7337236,"kb_used_data":5099140,"kb_used_omap":3903,"kb_used_meta":2234176,"kb_avail":87034604,"statfs":{"total":96636764160,"available":89123434496,"internally_reserved":0,"allocated":5221519360,"data_stored":55739801338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3997267,"internal_metadata":2287796653},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,3,7,4,6,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":3.1899999999999999,"5min":3.54,"15min":3.1699999999999999},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":14.814,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":3.0369999999999999,"5min":2.6829999999999998,"15min":2.77},"min":{"1min":0.255,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":13.880000000000001,"5min":24.917000000000002,"15min":24.917000000000002},"last":3.5640000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":2.3180000000000001,"15min":2.1019999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19400000000000001},"max":{"1min":11.691000000000001,"5min":13.358000000000001,"15min":21.326000000000001},"last":3.8340000000000001},{"interface":"front","average":{"1min":2.177,"5min":2.5579999999999998,"15min":2.7450000000000001},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":8.0909999999999993,"5min":24.867999999999999,"15min":24.867999999999999},"last":1.0920000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3215618048,"data_stored":36176469093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5218586624,"data_stored":55707841809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44305,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2002776064,"data_stored":19551508652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:26:01.513 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:01.513 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:01.513 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:01.513 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:01.514 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:26:01.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:01.910+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f0340 2023-10-03T18:26:01.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:01.910+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f08f0 2023-10-03T18:26:01.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:01.982+0000 7fc515f98700 -1 reset not still connected to 0x5622352af1e0 2023-10-03T18:26:02.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.029+0000 7fc515f98700 -1 reset not still connected to 0x56223b924d00 2023-10-03T18:26:02.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.088+0000 7fc515f98700 -1 reset not still connected to 0x56221c33d930 2023-10-03T18:26:02.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.114+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f0340 2023-10-03T18:26:02.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.114+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f08f0 2023-10-03T18:26:02.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.127+0000 7fc515f98700 -1 reset not still connected to 0x56222f369790 2023-10-03T18:26:02.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.153+0000 7fc515f98700 -1 reset not still connected to 0x5622352af1e0 2023-10-03T18:26:02.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.189+0000 7fc515f98700 -1 reset not still connected to 0x56223b924d00 2023-10-03T18:26:02.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.527+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f0340 2023-10-03T18:26:02.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.527+0000 7fc515f98700 -1 reset not still connected to 0x56221c7f08f0 2023-10-03T18:26:02.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.527+0000 7fc515f98700 -1 reset not still connected to 0x5622291fb450 2023-10-03T18:26:02.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.527+0000 7fc515f98700 -1 reset not still connected to 0x56222f369790 2023-10-03T18:26:02.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.611+0000 7fc515f98700 -1 reset not still connected to 0x5622352af1e0 2023-10-03T18:26:02.660 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:02.654+0000 7fc515f98700 -1 reset not still connected to 0x56223b924d00 2023-10-03T18:26:03.300 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:03.298+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:26:03.306 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:03.304+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:26:03.306 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:03.304+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:03.306 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:03.304+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:03.976 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:03.975+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:26:04.073 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:04.071+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:04.073 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:04.071+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:04.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.631+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:04.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.631+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:04.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.648+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:04.653 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.652+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:04.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.747+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:04.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:04.780+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:05.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.355+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:05.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.355+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:05.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.355+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:05.458 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.457+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:05.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.523+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:05.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.523+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:05.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:05.523+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:07.158 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:07.151+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:07.392 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:07.151+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:08.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.703+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:08.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.739+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:08.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:08.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:08.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:08.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:08.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:08.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:08.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:08.914+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:11.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.805+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:11.899 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.896+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:11.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.897+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:11.914 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.897+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:11.977 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:12.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:12.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:12.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:12.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:12.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.972+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:12.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:11.973+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:12.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.059+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:12.073 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.071+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:12.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.092+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:12.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.135+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:12.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.135+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:12.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.193+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:12.244 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.243+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:12.296 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.295+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:12.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.305+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:12.379 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.377+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:12.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.378+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:12.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.378+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:12.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.424+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:12.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.579+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:12.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.610+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:12.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.610+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:12.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.636+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:12.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.636+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:12.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.852+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:12.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.852+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:12.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.852+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:12.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.852+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:12.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.852+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:12.911 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.909+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:12.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.916+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:12.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.960+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:12.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:12.960+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:13.011 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:13.009+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:13.012 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:13.009+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:13.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:13.012+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:15.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.148+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:15.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.187+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:15.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.212+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:15.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.244+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:15.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.253+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:15.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.253+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:15.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.310+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:15.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.400+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:15.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:15.400+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:16.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:16.358+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:16.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:16.575+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:16.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:16.575+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:18.019 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.018+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:18.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.083+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:18.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.108+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:18.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.108+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:18.140 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.139+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:18.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.152+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:18.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.152+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:18.176 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.175+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:18.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.185+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:18.202 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.202+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:18.312 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.310+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:18.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.354+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:18.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.419+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:18.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.438+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:18.439 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.438+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:18.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.617+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:18.619 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.618+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:18.619 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.618+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:18.619 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:18.618+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:21.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:26:21.844 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:21.843+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:26:21.845 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:21.844+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:21.854 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:21.852+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:22.217 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:26:22.226 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:26:22.244 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1963,"stamp":"2023-10-03T18:26:21.343710+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23735461681,"num_objects":65128,"num_object_clones":59862,"num_object_copies":130256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65128,"num_whiteouts":0,"num_read":1414428,"num_read_kb":5130570,"num_write":1283520,"num_write_kb":5117903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80597,"ondisk_log_size":80597,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15578704,"kb_used_data":10237940,"kb_used_omap":7395,"kb_used_meta":5333340,"kb_avail":267536816,"statfs":{"total":289910292480,"available":273957699584,"internally_reserved":0,"allocated":10483650560,"data_stored":114391457379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7573456,"internal_metadata":5461340208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,5,13,8,15,16],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6479872,"num_objects":584,"num_object_clones":584,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":0,"num_read":1676,"num_read_kb":6680,"num_write":1554,"num_write_kb":6216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003980"},"pg_stats":[{"pgid":"2.7","version":"364'192015","reported_seq":340022,"reported_epoch":364,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:16.238058+0000","last_change":"2023-10-03T18:26:14.216348+0000","last_active":"2023-10-03T18:26:16.238058+0000","last_peered":"2023-10-03T18:26:16.238058+0000","last_clean":"2023-10-03T18:26:16.238058+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:26:16.238058+0000","last_undegraded":"2023-10-03T18:26:16.238058+0000","last_fullsized":"2023-10-03T18:26:16.238058+0000","mapping_epoch":12,"log_start":"317'181947","ondisk_log_start":"317'181947","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10068,"log_dups_size":0,"ondisk_log_size":10068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":286,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:12:17.033341+0000","scrub_duration":505.38444223300002,"objects_trimmed":12,"snaptrim_duration":1.6347578780000001,"stat_sum":{"num_bytes":2956325299,"num_objects":11536,"num_object_clones":10886,"num_object_copies":23072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11536,"num_whiteouts":0,"num_read":172457,"num_read_kb":638135,"num_write":159384,"num_write_kb":635721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"175","length":"1"}]},{"pgid":"2.6","version":"364'177860","reported_seq":315296,"reported_epoch":364,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:16.265348+0000","last_change":"2023-10-03T18:26:15.273889+0000","last_active":"2023-10-03T18:26:16.265348+0000","last_peered":"2023-10-03T18:26:16.265348+0000","last_clean":"2023-10-03T18:26:16.265348+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:26:16.265348+0000","last_undegraded":"2023-10-03T18:26:16.265348+0000","last_fullsized":"2023-10-03T18:26:16.265348+0000","mapping_epoch":12,"log_start":"311'167819","ondisk_log_start":"311'167819","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":201,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":2.1141623919999999,"stat_sum":{"num_bytes":2731651695,"num_objects":8247,"num_object_clones":7643,"num_object_copies":16494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8247,"num_whiteouts":0,"num_read":160007,"num_read_kb":588668,"num_write":147290,"num_write_kb":587934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"9"}]},{"pgid":"2.5","version":"364'213518","reported_seq":375600,"reported_epoch":364,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:16.185416+0000","last_change":"2023-10-03T18:26:15.256216+0000","last_active":"2023-10-03T18:26:16.185416+0000","last_peered":"2023-10-03T18:26:16.185416+0000","last_clean":"2023-10-03T18:26:16.185416+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:26:16.185416+0000","last_undegraded":"2023-10-03T18:26:16.185416+0000","last_fullsized":"2023-10-03T18:26:16.185416+0000","mapping_epoch":12,"log_start":"318'203420","ondisk_log_start":"318'203420","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":230,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":2.675753689,"stat_sum":{"num_bytes":3204141810,"num_objects":10653,"num_object_clones":9941,"num_object_copies":21306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10653,"num_whiteouts":0,"num_read":196981,"num_read_kb":696971,"num_write":173655,"num_write_kb":691384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"}]},{"pgid":"2.0","version":"364'206847","reported_seq":360273,"reported_epoch":364,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:18.756988+0000","last_change":"2023-10-03T18:26:17.873674+0000","last_active":"2023-10-03T18:26:18.756988+0000","last_peered":"2023-10-03T18:26:18.756988+0000","last_clean":"2023-10-03T18:26:18.756988+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:26:18.756988+0000","last_undegraded":"2023-10-03T18:26:18.756988+0000","last_fullsized":"2023-10-03T18:26:18.756988+0000","mapping_epoch":12,"log_start":"330'196759","ondisk_log_start":"330'196759","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":205,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:27:54.612837+0000","scrub_duration":734.24538164700004,"objects_trimmed":71,"snaptrim_duration":5.3552829790000001,"stat_sum":{"num_bytes":3095552757,"num_objects":9511,"num_object_clones":8824,"num_object_copies":19022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9511,"num_whiteouts":0,"num_read":179519,"num_read_kb":665441,"num_write":167531,"num_write_kb":668316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":528,"reported_epoch":363,"state":"active+clean","last_fresh":"2023-10-03T18:26:09.668990+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:26:09.668990+0000","last_peered":"2023-10-03T18:26:09.668990+0000","last_clean":"2023-10-03T18:26:09.668990+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:26:09.668990+0000","last_undegraded":"2023-10-03T18:26:09.668990+0000","last_fullsized":"2023-10-03T18:26:09.668990+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"364'205458","reported_seq":361223,"reported_epoch":364,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:16.265193+0000","last_change":"2023-10-03T18:26:14.254177+0000","last_active":"2023-10-03T18:26:16.265193+0000","last_peered":"2023-10-03T18:26:16.265193+0000","last_clean":"2023-10-03T18:26:16.265193+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:26:16.265193+0000","last_undegraded":"2023-10-03T18:26:16.265193+0000","last_fullsized":"2023-10-03T18:26:16.265193+0000","mapping_epoch":12,"log_start":"317'195432","ondisk_log_start":"317'195432","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":222,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":0.22922804999999999,"stat_sum":{"num_bytes":3148296932,"num_objects":10159,"num_object_clones":9462,"num_object_copies":20318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10159,"num_whiteouts":0,"num_read":182751,"num_read_kb":676855,"num_write":169860,"num_write_kb":677557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"}]},{"pgid":"2.1","version":"364'201384","reported_seq":342842,"reported_epoch":364,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:18.778637+0000","last_change":"2023-10-03T18:26:18.236689+0000","last_active":"2023-10-03T18:26:18.778637+0000","last_peered":"2023-10-03T18:26:18.778637+0000","last_clean":"2023-10-03T18:26:18.778637+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:26:18.778637+0000","last_undegraded":"2023-10-03T18:26:18.778637+0000","last_fullsized":"2023-10-03T18:26:18.778637+0000","mapping_epoch":12,"log_start":"318'191311","ondisk_log_start":"318'191311","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:47:42.413851+0000","scrub_duration":424.08306342600002,"objects_trimmed":88,"snaptrim_duration":5.717826101,"stat_sum":{"num_bytes":2868597207,"num_objects":3098,"num_object_clones":2466,"num_object_copies":6196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3098,"num_whiteouts":0,"num_read":172271,"num_read_kb":619143,"num_write":155063,"num_write_kb":617821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"7"}]},{"pgid":"2.2","version":"364'201630","reported_seq":348203,"reported_epoch":364,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:20.176723+0000","last_change":"2023-10-03T18:26:17.889391+0000","last_active":"2023-10-03T18:26:20.176723+0000","last_peered":"2023-10-03T18:26:20.176723+0000","last_clean":"2023-10-03T18:26:20.176723+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:26:20.176723+0000","last_undegraded":"2023-10-03T18:26:20.176723+0000","last_fullsized":"2023-10-03T18:26:20.176723+0000","mapping_epoch":12,"log_start":"318'191568","ondisk_log_start":"318'191568","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:20:51.288427+0000","scrub_duration":370.574046178,"objects_trimmed":105,"snaptrim_duration":5.3493222109999996,"stat_sum":{"num_bytes":2871997442,"num_objects":3091,"num_object_clones":2444,"num_object_copies":6182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3091,"num_whiteouts":0,"num_read":181789,"num_read_kb":630332,"num_write":156384,"num_write_kb":622267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"7"}]},{"pgid":"2.4","version":"364'187859","reported_seq":331361,"reported_epoch":364,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:16.203018+0000","last_change":"2023-10-03T18:26:11.985526+0000","last_active":"2023-10-03T18:26:16.203018+0000","last_peered":"2023-10-03T18:26:16.203018+0000","last_clean":"2023-10-03T18:26:16.203018+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:26:16.203018+0000","last_undegraded":"2023-10-03T18:26:16.203018+0000","last_fullsized":"2023-10-03T18:26:16.203018+0000","mapping_epoch":12,"log_start":"316'177817","ondisk_log_start":"316'177817","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":207,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.3237206480000001,"stat_sum":{"num_bytes":2858308171,"num_objects":8831,"num_object_clones":8196,"num_object_copies":17662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8831,"num_whiteouts":0,"num_read":168547,"num_read_kb":614934,"num_write":154216,"num_write_kb":615009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"175","length":"3"},{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23734871313,"num_objects":65126,"num_object_clones":59862,"num_object_copies":130252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65126,"num_whiteouts":0,"num_read":1414322,"num_read_kb":5130479,"num_write":1283383,"num_write_kb":5116009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10471473152,"data_stored":114484275746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80770,"internal_metadata":0},"log_size":80498,"ondisk_log_size":80498,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739011,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3002556,"kb_used_data":1966580,"kb_used_omap":1375,"kb_used_meta":1034592,"kb_avail":91369284,"statfs":{"total":96636764160,"available":93562146816,"internally_reserved":0,"allocated":2013777920,"data_stored":19966898779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1059422340},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,3,3,2,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.665,"5min":2.4079999999999999,"15min":2.6040000000000001},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.16600000000000001},"max":{"1min":12.737,"5min":15.964,"15min":18.321000000000002},"last":1.95},{"interface":"front","average":{"1min":3.2109999999999999,"5min":2.141,"15min":2.258},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":18.309999999999999,"5min":18.309999999999999,"15min":30.128},"last":0.30599999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:25:22 2023","interfaces":[{"interface":"back","average":{"1min":2.6190000000000002,"5min":2.383,"15min":2.395},"min":{"1min":0.25,"5min":0.25,"15min":0.21199999999999999},"max":{"1min":12.368,"5min":12.368,"15min":18.768999999999998},"last":2.1200000000000001},{"interface":"front","average":{"1min":3.0139999999999998,"5min":2.423,"15min":2.323},"min":{"1min":0.25,"5min":0.214,"15min":0.19700000000000001},"max":{"1min":12.449,"5min":12.449,"15min":18.751999999999999},"last":2.137}]}]},{"osd":0,"up_from":8,"seq":34359739010,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5066908,"kb_used_data":3156244,"kb_used_omap":2116,"kb_used_meta":1908539,"kb_avail":89304932,"statfs":{"total":96636764160,"available":91448250368,"internally_reserved":0,"allocated":3231993856,"data_stored":37385902636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":1954344447},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":5.3380000000000001,"5min":3.0470000000000002,"15min":3.226},"min":{"1min":0.25900000000000001,"5min":0.17499999999999999,"15min":0.14799999999999999},"max":{"1min":45.886000000000003,"5min":45.886000000000003,"15min":45.886000000000003},"last":2.7149999999999999},{"interface":"front","average":{"1min":5.2290000000000001,"5min":3.1429999999999998,"15min":3.2789999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.34999999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.7869999999999999,"15min":1.903},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":8.9220000000000006,"5min":12.752000000000001,"15min":13.209},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.784,"5min":2.242,"15min":2.1379999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":5.8570000000000002,"5min":13.459,"15min":16.033999999999999},"last":0.311}]}]},{"osd":1,"up_from":8,"seq":34359738983,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7509240,"kb_used_data":5115116,"kb_used_omap":3903,"kb_used_meta":2390208,"kb_avail":86862600,"statfs":{"total":96636764160,"available":88947302400,"internally_reserved":0,"allocated":5237878784,"data_stored":57038655964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3997267,"internal_metadata":2447573421},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,7,6,12,15],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":3.1899999999999999,"5min":3.54,"15min":3.1699999999999999},"min":{"1min":0.19500000000000001,"5min":0.184,"15min":0.182},"max":{"1min":14.814,"5min":30.896999999999998,"15min":30.896999999999998},"last":2.677},{"interface":"front","average":{"1min":3.0369999999999999,"5min":2.6829999999999998,"15min":2.77},"min":{"1min":0.255,"5min":0.222,"15min":0.17799999999999999},"max":{"1min":13.880000000000001,"5min":24.917000000000002,"15min":24.917000000000002},"last":3.3380000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:25:31 2023","interfaces":[{"interface":"back","average":{"1min":1.982,"5min":2.3180000000000001,"15min":2.1019999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.19400000000000001},"max":{"1min":11.691000000000001,"5min":13.358000000000001,"15min":21.326000000000001},"last":0.30199999999999999},{"interface":"front","average":{"1min":2.177,"5min":2.5579999999999998,"15min":2.7450000000000001},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":8.0909999999999993,"5min":24.867999999999999,"15min":24.867999999999999},"last":0.38200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3227504640,"data_stored":37396413793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5234081792,"data_stored":57108817169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":44305,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2009886720,"data_stored":19979044784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:26:22.245 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:22.245 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:22.246 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:22.246 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:22.246 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:26:22.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.507+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:22.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.566+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:22.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.566+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:22.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.578+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:22.640 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.639+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:22.744 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.743+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:22.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:22.796+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:26.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:25.998+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:26.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.111+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:26.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.111+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:26.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.111+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:26.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.112+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:26.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.115+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:26.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:26.115+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:26.601 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:26.599+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:27.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:27.950+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:27.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:27.950+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:27.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:27.950+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:27.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:27.963+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:27.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:27.963+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:28.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:28.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:28.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:28.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:28.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:28.025 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.022+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:28.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.074+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:28.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.137+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:28.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.256+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:28.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.358+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:28.363 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.358+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:28.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.370+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:28.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.469+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:28.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.469+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:28.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.469+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:28.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.469+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:28.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:28.469+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:29.294 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.293+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:29.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.316+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:29.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.415+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:29.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.496+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:29.515 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:29.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:29.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:29.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:29.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:29.516 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:29.514+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:30.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.469+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:30.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.469+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:30.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.547+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:30.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.547+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:30.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.578+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:30.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.578+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:30.580 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.578+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:30.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.600+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:30.613 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:30.604+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:31.375 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.374+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:31.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.374+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:31.376 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.374+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:31.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.381+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:31.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.449+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:31.638 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.637+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:31.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.649+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:31.650 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.650+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:31.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.750+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:31.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.799+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:31.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.807+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:31.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.807+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:31.808 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.807+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:31.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.819+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:31.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.849+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:31.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.876+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:31.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.953+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:31.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:31.953+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:32.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:32.020+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:32.058 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:32.055+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:32.069 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:32.068+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:33.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.363+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:33.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.413+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:33.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.421+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:33.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.497+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:33.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.512+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:33.626 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.621+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:33.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.628+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:33.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.628+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:33.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.629+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:33.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.629+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:33.698 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:33.698+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:34.029 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:34.026+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:26:34.035 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:34.035+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:34.035 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:34.035+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:36.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.612+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:36.654 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.648+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:36.708 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.699+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:36.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.752+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:36.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.781+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:36.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.781+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:36.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.781+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:36.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.782+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:36.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.968+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:36.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:36.995+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:37.782 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:37.774+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:38.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.729+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:38.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:38.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:38.771 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.770+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:38.841 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.841+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:38.921 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.919+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:38.921 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.919+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:38.992 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.991+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:38.993 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:38.991+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:39.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:39.038+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:42.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:26:43.185 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:26:43.228 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:26:43.238 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1977,"stamp":"2023-10-03T18:26:42.515402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23758321457,"num_objects":65678,"num_object_clones":60412,"num_object_copies":131356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65678,"num_whiteouts":0,"num_read":1421284,"num_read_kb":5154125,"num_write":1288947,"num_write_kb":5139611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80294,"ondisk_log_size":80294,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14661980,"kb_used_data":10281540,"kb_used_omap":8419,"kb_used_meta":4371996,"kb_avail":268453540,"statfs":{"total":289910292480,"available":274896424960,"internally_reserved":0,"allocated":10528296960,"data_stored":115772251651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8622000,"internal_metadata":4476923984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,1,1,6,17,31,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11685888,"num_objects":-2,"num_object_clones":-2,"num_object_copies":-4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":0,"num_read":4060,"num_read_kb":12419,"num_write":2796,"num_write_kb":11184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007830"},"pg_stats":[{"pgid":"2.7","version":"367'193274","reported_seq":341689,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:38.840090+0000","last_change":"2023-10-03T18:26:37.286004+0000","last_active":"2023-10-03T18:26:38.840090+0000","last_peered":"2023-10-03T18:26:38.840090+0000","last_clean":"2023-10-03T18:26:38.840090+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:26:38.840090+0000","last_undegraded":"2023-10-03T18:26:38.840090+0000","last_fullsized":"2023-10-03T18:26:38.840090+0000","mapping_epoch":12,"log_start":"319'183254","ondisk_log_start":"319'183254","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":283,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:12:17.033341+0000","scrub_duration":505.38444223300002,"objects_trimmed":48,"snaptrim_duration":1.6717756110000002,"stat_sum":{"num_bytes":2959401395,"num_objects":11688,"num_object_clones":11038,"num_object_copies":23376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11688,"num_whiteouts":0,"num_read":173271,"num_read_kb":640993,"num_write":160104,"num_write_kb":638601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"179","length":"1"}]},{"pgid":"2.6","version":"367'178989","reported_seq":316876,"reported_epoch":367,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:38.693858+0000","last_change":"2023-10-03T18:26:35.194558+0000","last_active":"2023-10-03T18:26:38.693858+0000","last_peered":"2023-10-03T18:26:38.693858+0000","last_clean":"2023-10-03T18:26:38.693858+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:26:38.693858+0000","last_undegraded":"2023-10-03T18:26:38.693858+0000","last_fullsized":"2023-10-03T18:26:38.693858+0000","mapping_epoch":12,"log_start":"317'168925","ondisk_log_start":"317'168925","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":197,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":33,"snaptrim_duration":2.6527149429999999,"stat_sum":{"num_bytes":2734367343,"num_objects":8379,"num_object_clones":7775,"num_object_copies":16758,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8379,"num_whiteouts":0,"num_read":160776,"num_read_kb":591497,"num_write":147941,"num_write_kb":590538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"3"}]},{"pgid":"2.5","version":"367'214833","reported_seq":377439,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:38.650255+0000","last_change":"2023-10-03T18:26:36.924723+0000","last_active":"2023-10-03T18:26:38.650255+0000","last_peered":"2023-10-03T18:26:38.650255+0000","last_clean":"2023-10-03T18:26:38.650255+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:26:38.650255+0000","last_undegraded":"2023-10-03T18:26:38.650255+0000","last_fullsized":"2023-10-03T18:26:38.650255+0000","mapping_epoch":12,"log_start":"322'204825","ondisk_log_start":"322'204825","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10008,"log_dups_size":0,"ondisk_log_size":10008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":228,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":0.14160999299999999,"stat_sum":{"num_bytes":3207418610,"num_objects":10844,"num_object_clones":10132,"num_object_copies":21688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10844,"num_whiteouts":0,"num_read":197978,"num_read_kb":700324,"num_write":174427,"num_write_kb":694472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"}]},{"pgid":"2.0","version":"367'208282","reported_seq":362138,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:38.861595+0000","last_change":"2023-10-03T18:26:38.454882+0000","last_active":"2023-10-03T18:26:38.861595+0000","last_peered":"2023-10-03T18:26:38.861595+0000","last_clean":"2023-10-03T18:26:38.861595+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:26:38.861595+0000","last_undegraded":"2023-10-03T18:26:38.861595+0000","last_fullsized":"2023-10-03T18:26:38.861595+0000","mapping_epoch":12,"log_start":"332'198267","ondisk_log_start":"332'198267","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":196,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:27:54.612837+0000","scrub_duration":734.24538164700004,"objects_trimmed":88,"snaptrim_duration":5.9116723770000004,"stat_sum":{"num_bytes":3098194677,"num_objects":9468,"num_object_clones":8781,"num_object_copies":18936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9468,"num_whiteouts":0,"num_read":180361,"num_read_kb":668423,"num_write":168174,"num_write_kb":670888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"4"}]},{"pgid":"1.0","version":"88'99","reported_seq":531,"reported_epoch":366,"state":"active+clean","last_fresh":"2023-10-03T18:26:27.623079+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:26:27.623079+0000","last_peered":"2023-10-03T18:26:27.623079+0000","last_clean":"2023-10-03T18:26:27.623079+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:26:27.623079+0000","last_undegraded":"2023-10-03T18:26:27.623079+0000","last_fullsized":"2023-10-03T18:26:27.623079+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"367'206737","reported_seq":362997,"reported_epoch":367,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:38.635963+0000","last_change":"2023-10-03T18:26:37.283538+0000","last_active":"2023-10-03T18:26:38.635963+0000","last_peered":"2023-10-03T18:26:38.635963+0000","last_clean":"2023-10-03T18:26:38.635963+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:26:38.635963+0000","last_undegraded":"2023-10-03T18:26:38.635963+0000","last_fullsized":"2023-10-03T18:26:38.635963+0000","mapping_epoch":12,"log_start":"320'196735","ondisk_log_start":"320'196735","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10002,"log_dups_size":0,"ondisk_log_size":10002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":218,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":4.756070791,"stat_sum":{"num_bytes":3151307492,"num_objects":10343,"num_object_clones":9646,"num_object_copies":20686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10343,"num_whiteouts":0,"num_read":183667,"num_read_kb":680137,"num_write":170587,"num_write_kb":680465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"2"}]},{"pgid":"2.1","version":"367'202845","reported_seq":344666,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:38.860410+0000","last_change":"2023-10-03T18:26:38.110709+0000","last_active":"2023-10-03T18:26:38.860410+0000","last_peered":"2023-10-03T18:26:38.860410+0000","last_clean":"2023-10-03T18:26:38.860410+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:26:38.860410+0000","last_undegraded":"2023-10-03T18:26:38.860410+0000","last_fullsized":"2023-10-03T18:26:38.860410+0000","mapping_epoch":12,"log_start":"330'192821","ondisk_log_start":"330'192821","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:47:42.413851+0000","scrub_duration":424.08306342600002,"objects_trimmed":100,"snaptrim_duration":5.5859421569999999,"stat_sum":{"num_bytes":2871271895,"num_objects":2986,"num_object_clones":2354,"num_object_copies":5972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2986,"num_whiteouts":0,"num_read":173043,"num_read_kb":621774,"num_write":155688,"num_write_kb":620321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"}]},{"pgid":"2.2","version":"367'203033","reported_seq":350002,"reported_epoch":367,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:26:40.341163+0000","last_change":"2023-10-03T18:26:39.200794+0000","last_active":"2023-10-03T18:26:40.341163+0000","last_peered":"2023-10-03T18:26:40.341163+0000","last_clean":"2023-10-03T18:26:40.341163+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:26:40.341163+0000","last_undegraded":"2023-10-03T18:26:40.341163+0000","last_fullsized":"2023-10-03T18:26:40.341163+0000","mapping_epoch":12,"log_start":"330'192974","ondisk_log_start":"330'192974","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:20:51.288427+0000","scrub_duration":370.574046178,"objects_trimmed":119,"snaptrim_duration":6.6486662670000003,"stat_sum":{"num_bytes":2874524674,"num_objects":2998,"num_object_clones":2351,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":182684,"num_read_kb":632991,"num_write":156986,"num_write_kb":624675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"}]},{"pgid":"2.4","version":"367'189026","reported_seq":333012,"reported_epoch":367,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:26:38.640684+0000","last_change":"2023-10-03T18:26:36.907008+0000","last_active":"2023-10-03T18:26:38.640684+0000","last_peered":"2023-10-03T18:26:38.640684+0000","last_clean":"2023-10-03T18:26:38.640684+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:26:38.640684+0000","last_undegraded":"2023-10-03T18:26:38.640684+0000","last_fullsized":"2023-10-03T18:26:38.640684+0000","mapping_epoch":12,"log_start":"318'179023","ondisk_log_start":"318'179023","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":204,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":4.3795568899999999,"stat_sum":{"num_bytes":2861245003,"num_objects":8970,"num_object_clones":8335,"num_object_copies":17940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8970,"num_whiteouts":0,"num_read":169398,"num_read_kb":617895,"num_write":154903,"num_write_kb":617757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"179","length":"2"},{"start":"17c","length":"2"},{"start":"186","length":"a"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23757731089,"num_objects":65676,"num_object_clones":60412,"num_object_copies":131352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65676,"num_whiteouts":0,"num_read":1421178,"num_read_kb":5154034,"num_write":1288810,"num_write_kb":5137717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10515759104,"data_stored":115748973090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68668,"internal_metadata":0},"log_size":80195,"ondisk_log_size":80195,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739015,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3072704,"kb_used_data":1974264,"kb_used_omap":1375,"kb_used_meta":1097056,"kb_avail":91299136,"statfs":{"total":96636764160,"available":93490315264,"internally_reserved":0,"allocated":2021646336,"data_stored":20049502779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1123385476},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,4,3,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":2.411,"5min":2.46,"15min":2.5470000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.922000000000001,"5min":15.964,"15min":18.321000000000002},"last":11.4},{"interface":"front","average":{"1min":4.2839999999999998,"5min":2.653,"15min":2.4510000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":34.176000000000002,"5min":34.176000000000002,"15min":34.176000000000002},"last":9.1500000000000004}]},{"osd":1,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2110000000000003,"5min":2.609,"15min":2.508},"min":{"1min":0.27900000000000003,"5min":0.25,"15min":0.221},"max":{"1min":15.85,"5min":15.85,"15min":18.768999999999998},"last":4.8929999999999998},{"interface":"front","average":{"1min":3.8519999999999999,"5min":2.661,"15min":2.4079999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":18.751999999999999},"last":0.24399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739014,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5183812,"kb_used_data":3169020,"kb_used_omap":2116,"kb_used_meta":2012667,"kb_avail":89188028,"statfs":{"total":96636764160,"available":91328540672,"internally_reserved":0,"allocated":3245076480,"data_stored":37840173580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":2060971519},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,1,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":5.3380000000000001,"5min":3.0470000000000002,"15min":3.226},"min":{"1min":0.25900000000000001,"5min":0.17499999999999999,"15min":0.14799999999999999},"max":{"1min":45.886000000000003,"5min":45.886000000000003,"15min":45.886000000000003},"last":0.219},{"interface":"front","average":{"1min":5.2290000000000001,"5min":3.1429999999999998,"15min":3.2789999999999999},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.20699999999999999},"max":{"1min":45.555,"5min":45.555,"15min":45.555},"last":0.20100000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:25:43 2023","interfaces":[{"interface":"back","average":{"1min":2.395,"5min":1.7869999999999999,"15min":1.903},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":8.9220000000000006,"5min":12.752000000000001,"15min":13.209},"last":10.582000000000001},{"interface":"front","average":{"1min":1.784,"5min":2.242,"15min":2.1379999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":5.8570000000000002,"5min":13.459,"15min":16.033999999999999},"last":0.246}]}]},{"osd":1,"up_from":8,"seq":34359738987,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6405464,"kb_used_data":5138256,"kb_used_omap":4927,"kb_used_meta":1262272,"kb_avail":87966376,"statfs":{"total":96636764160,"available":90077569024,"internally_reserved":0,"allocated":5261574144,"data_stored":57882575292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1292566989},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,0,1,0,2,13,29],"upper_bound":256},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.266,"5min":3.218,"15min":3.1280000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.3320000000000007,"5min":30.896999999999998,"15min":30.896999999999998},"last":0.76900000000000002},{"interface":"front","average":{"1min":2.3980000000000001,"5min":2.7639999999999998,"15min":2.831},"min":{"1min":0.24299999999999999,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":11.779,"5min":24.917000000000002,"15min":24.917000000000002},"last":0.83699999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":2.4329999999999998,"15min":2.1040000000000001},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":17.084,"5min":17.084,"15min":17.084},"last":8.9079999999999995},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.556,"15min":2.7519999999999998},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":11.736000000000001,"5min":24.867999999999999,"15min":24.867999999999999},"last":11.779}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3240734720,"data_stored":37827439969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5257252864,"data_stored":57875625233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2017771520,"data_stored":20045907888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:26:43.240 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:43.240 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:43.240 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:43.240 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:26:43.241 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:26:44.019 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:26:44.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:26:44.172 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:44.170+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:26:44.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:26:45.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.029+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:45.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.029+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:45.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.029+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:45.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.033+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:45.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.039+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:45.043 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:45.039+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:45.509 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:26:45.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:26:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:26:46.237 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:26:48.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.633+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:48.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.658+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:48.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.658+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:48.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.681+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:48.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.695+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:48.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:48.695+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:49.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.254+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:49.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.352+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:49.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.406+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:49.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.412+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:49.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.412+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:49.417 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.412+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:49.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.446+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:49.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.446+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:49.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.446+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:49.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.446+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:49.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:49.446+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:51.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.189+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:51.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.189+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:51.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.190+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:51.238 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.234+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:51.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.239+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:51.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.239+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:51.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.239+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:51.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.253+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:51.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:51.254+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:52.476 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:52.474+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:26:52.563 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:26:52.562+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:26:52.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.968+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:26:52.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.974+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:26:52.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.974+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:26:52.984 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.974+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:52.998 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.997+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:52.998 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:52.997+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:53.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:53.031+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:56.063 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.061+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:56.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.066+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:56.902 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.897+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:26:56.927 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.918+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.948+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.948+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:56.948+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:57.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:57.045+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:57.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:57.046+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:57.048 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:57.046+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:26:59.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.449+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:26:59.455 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.449+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:26:59.487 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.483+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:26:59.487 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.483+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:26:59.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.483+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:26:59.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.531+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:26:59.535 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:26:59.531+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:01.193 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:27:01.191+0000 7f9f21f20700 -1 reset not still connected to 0x5632ef289d40 2023-10-03T18:27:01.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:01.907+0000 7fc515f98700 -1 reset not still connected to 0x56220e08ac30 2023-10-03T18:27:02.039 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.038+0000 7fc515f98700 -1 reset not still connected to 0x562214d80270 2023-10-03T18:27:02.093 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.089+0000 7fc515f98700 -1 reset not still connected to 0x56221927d110 2023-10-03T18:27:02.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.569+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:02.574 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.569+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:02.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.880+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:02.882 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.880+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:02.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.900+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:02.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:02.972+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:03.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.040+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:03.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.156+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:03.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.156+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:03.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.156+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:03.310 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.308+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:03.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.333+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:03.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.333+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:03.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.394+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:03.396 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.394+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:03.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.402+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:03.975 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:03.972+0000 7fc515f98700 -1 reset not still connected to 0x56223b3a4340 2023-10-03T18:27:06.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:27:07.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:27:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:27:07.141 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":1999,"stamp":"2023-10-03T18:27:06.647718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23794075492,"num_objects":65504,"num_object_clones":60237,"num_object_copies":131008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65504,"num_whiteouts":0,"num_read":1432780,"num_read_kb":5190654,"num_write":1297713,"num_write_kb":5174126,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80593,"ondisk_log_size":80593,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14941908,"kb_used_data":10349864,"kb_used_omap":8419,"kb_used_meta":4583580,"kb_avail":268173612,"statfs":{"total":289910292480,"available":274609778688,"internally_reserved":0,"allocated":10598260736,"data_stored":115693376617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8622000,"internal_metadata":4693586000},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,11,10,18,17,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5922867,"num_objects":226,"num_object_clones":225,"num_object_copies":452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":226,"num_whiteouts":0,"num_read":3967,"num_read_kb":7784,"num_write":1601,"num_write_kb":5855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.102787"},"pg_stats":[{"pgid":"2.7","version":"376'194878","reported_seq":344246,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:05.535464+0000","last_change":"2023-10-03T18:27:05.327943+0000","last_active":"2023-10-03T18:27:05.535464+0000","last_peered":"2023-10-03T18:27:05.535464+0000","last_clean":"2023-10-03T18:27:05.535464+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:27:05.535464+0000","last_undegraded":"2023-10-03T18:27:05.535464+0000","last_fullsized":"2023-10-03T18:27:05.535464+0000","mapping_epoch":12,"log_start":"330'184862","ondisk_log_start":"330'184862","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":289,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:39:45.835669+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.62487115400000004,"stat_sum":{"num_bytes":2963886615,"num_objects":11694,"num_object_clones":11042,"num_object_copies":23388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11694,"num_whiteouts":0,"num_read":174637,"num_read_kb":645515,"num_write":161199,"num_write_kb":642928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"188","length":"1"}]},{"pgid":"2.6","version":"376'180415","reported_seq":319339,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:05.520425+0000","last_change":"2023-10-03T18:27:03.946373+0000","last_active":"2023-10-03T18:27:05.520425+0000","last_peered":"2023-10-03T18:27:05.520425+0000","last_clean":"2023-10-03T18:27:05.520425+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:27:05.520425+0000","last_undegraded":"2023-10-03T18:27:05.520425+0000","last_fullsized":"2023-10-03T18:27:05.520425+0000","mapping_epoch":12,"log_start":"321'170329","ondisk_log_start":"321'170329","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":205,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.121401334,"stat_sum":{"num_bytes":2738660000,"num_objects":8396,"num_object_clones":7791,"num_object_copies":16792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8396,"num_whiteouts":0,"num_read":162128,"num_read_kb":595969,"num_write":148978,"num_write_kb":594640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"7"}]},{"pgid":"2.5","version":"376'216505","reported_seq":380248,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:05.503957+0000","last_change":"2023-10-03T18:27:03.921061+0000","last_active":"2023-10-03T18:27:05.503957+0000","last_peered":"2023-10-03T18:27:05.503957+0000","last_clean":"2023-10-03T18:27:05.503957+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:27:05.503957+0000","last_undegraded":"2023-10-03T18:27:05.503957+0000","last_fullsized":"2023-10-03T18:27:05.503957+0000","mapping_epoch":12,"log_start":"332'206432","ondisk_log_start":"332'206432","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":235,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.28832978399999998,"stat_sum":{"num_bytes":3212288655,"num_objects":10876,"num_object_clones":10166,"num_object_copies":21752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10876,"num_whiteouts":0,"num_read":199705,"num_read_kb":705405,"num_write":175631,"num_write_kb":699176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"}]},{"pgid":"2.0","version":"375'210357","reported_seq":365064,"reported_epoch":375,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:03.993742+0000","last_change":"2023-10-03T18:27:03.494287+0000","last_active":"2023-10-03T18:27:03.993742+0000","last_peered":"2023-10-03T18:27:03.993742+0000","last_clean":"2023-10-03T18:27:03.993742+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:27:03.993742+0000","last_undegraded":"2023-10-03T18:27:03.993742+0000","last_fullsized":"2023-10-03T18:27:03.993742+0000","mapping_epoch":12,"log_start":"337'200275","ondisk_log_start":"337'200275","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":193,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:51:07.341063+0000","scrub_duration":734.24538164700004,"objects_trimmed":4,"snaptrim_duration":0.91102016799999996,"stat_sum":{"num_bytes":3102774006,"num_objects":9204,"num_object_clones":8517,"num_object_copies":18408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9204,"num_whiteouts":0,"num_read":181644,"num_read_kb":672814,"num_write":169298,"num_write_kb":675339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":540,"reported_epoch":375,"state":"active+clean","last_fresh":"2023-10-03T18:27:03.769901+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:27:03.769901+0000","last_peered":"2023-10-03T18:27:03.769901+0000","last_clean":"2023-10-03T18:27:03.769901+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:27:03.769901+0000","last_undegraded":"2023-10-03T18:27:03.769901+0000","last_fullsized":"2023-10-03T18:27:03.769901+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"376'208402","reported_seq":365686,"reported_epoch":376,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:05.475697+0000","last_change":"2023-10-03T18:27:00.482242+0000","last_active":"2023-10-03T18:27:05.475697+0000","last_peered":"2023-10-03T18:27:05.475697+0000","last_clean":"2023-10-03T18:27:05.475697+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:27:05.475697+0000","last_undegraded":"2023-10-03T18:27:05.475697+0000","last_fullsized":"2023-10-03T18:27:05.475697+0000","mapping_epoch":12,"log_start":"330'198338","ondisk_log_start":"330'198338","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10064,"log_dups_size":0,"ondisk_log_size":10064,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":226,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.9281608970000002,"stat_sum":{"num_bytes":3156222743,"num_objects":10377,"num_object_clones":9679,"num_object_copies":20754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10377,"num_whiteouts":0,"num_read":185108,"num_read_kb":684940,"num_write":171787,"num_write_kb":685208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"6"}]},{"pgid":"2.1","version":"375'204267","reported_seq":346987,"reported_epoch":375,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:04.003920+0000","last_change":"2023-10-03T18:27:02.824019+0000","last_active":"2023-10-03T18:27:04.003920+0000","last_peered":"2023-10-03T18:27:04.003920+0000","last_clean":"2023-10-03T18:27:04.003920+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:27:04.003920+0000","last_undegraded":"2023-10-03T18:27:04.003920+0000","last_fullsized":"2023-10-03T18:27:04.003920+0000","mapping_epoch":12,"log_start":"332'194227","ondisk_log_start":"332'194227","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:56:18.075172+0000","scrub_duration":424.08306342600002,"objects_trimmed":8,"snaptrim_duration":0.24574610199999999,"stat_sum":{"num_bytes":2875662958,"num_objects":3006,"num_object_clones":2371,"num_object_copies":6012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3006,"num_whiteouts":0,"num_read":174324,"num_read_kb":625837,"num_write":156755,"num_write_kb":624542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"1"}]},{"pgid":"2.2","version":"376'204476","reported_seq":352513,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:05.564403+0000","last_change":"2023-10-03T18:27:03.521307+0000","last_active":"2023-10-03T18:27:05.564403+0000","last_peered":"2023-10-03T18:27:05.564403+0000","last_clean":"2023-10-03T18:27:05.564403+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:27:05.564403+0000","last_undegraded":"2023-10-03T18:27:05.564403+0000","last_fullsized":"2023-10-03T18:27:05.564403+0000","mapping_epoch":12,"log_start":"332'194376","ondisk_log_start":"332'194376","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10100,"log_dups_size":0,"ondisk_log_size":10100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:27:48.989229+0000","scrub_duration":370.574046178,"objects_trimmed":10,"snaptrim_duration":0.92856723299999999,"stat_sum":{"num_bytes":2878771974,"num_objects":2994,"num_object_clones":2352,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":184333,"num_read_kb":637654,"num_write":158042,"num_write_kb":628760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"1"}]},{"pgid":"2.4","version":"376'190460","reported_seq":335460,"reported_epoch":376,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:05.505437+0000","last_change":"2023-10-03T18:27:05.355786+0000","last_active":"2023-10-03T18:27:05.505437+0000","last_peered":"2023-10-03T18:27:05.505437+0000","last_clean":"2023-10-03T18:27:05.505437+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:27:05.505437+0000","last_undegraded":"2023-10-03T18:27:05.505437+0000","last_fullsized":"2023-10-03T18:27:05.505437+0000","mapping_epoch":12,"log_start":"324'180427","ondisk_log_start":"324'180427","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10033,"log_dups_size":0,"ondisk_log_size":10033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":211,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":2.257461148,"stat_sum":{"num_bytes":2865218173,"num_objects":8955,"num_object_clones":8319,"num_object_copies":17910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8955,"num_whiteouts":0,"num_read":170795,"num_read_kb":622429,"num_write":155886,"num_write_kb":621639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"188","length":"8"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23793485124,"num_objects":65502,"num_object_clones":60237,"num_object_copies":131004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65502,"num_whiteouts":0,"num_read":1432674,"num_read_kb":5190563,"num_write":1297576,"num_write_kb":5172232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10585284608,"data_stored":115782527624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68668,"internal_metadata":0},"log_size":80494,"ondisk_log_size":80494,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739020,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3140024,"kb_used_data":1987428,"kb_used_omap":1375,"kb_used_meta":1151200,"kb_avail":91231816,"statfs":{"total":96636764160,"available":93421379584,"internally_reserved":0,"allocated":2035126272,"data_stored":19875396537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1178828932},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,6,8,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":2.411,"5min":2.46,"15min":2.5470000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.922000000000001,"5min":15.964,"15min":18.321000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":4.2839999999999998,"5min":2.653,"15min":2.4510000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":34.176000000000002,"5min":34.176000000000002,"15min":34.176000000000002},"last":2.2610000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2110000000000003,"5min":2.609,"15min":2.508},"min":{"1min":0.27900000000000003,"5min":0.25,"15min":0.221},"max":{"1min":15.85,"5min":15.85,"15min":18.768999999999998},"last":2.3090000000000002},{"interface":"front","average":{"1min":3.8519999999999999,"5min":2.661,"15min":2.4079999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":18.751999999999999},"last":0.42999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739019,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5253836,"kb_used_data":3189056,"kb_used_omap":2116,"kb_used_meta":2062651,"kb_avail":89118004,"statfs":{"total":96636764160,"available":91256836096,"internally_reserved":0,"allocated":3265593344,"data_stored":37928894567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":2112155135},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:26:43 2023","interfaces":[{"interface":"back","average":{"1min":2.5430000000000001,"5min":3.1960000000000002,"15min":3.1869999999999998},"min":{"1min":0.188,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":13.396000000000001,"5min":45.886000000000003,"15min":45.886000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":2.617,"5min":3.0960000000000001,"15min":3.3069999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":13.538,"5min":45.555,"15min":45.555},"last":0.96299999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:26:43 2023","interfaces":[{"interface":"back","average":{"1min":2.4860000000000002,"5min":2.0859999999999999,"15min":1.978},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":11.252000000000001,"5min":12.752000000000001,"15min":13.209},"last":0.34999999999999998},{"interface":"front","average":{"1min":3.5670000000000002,"5min":2.4390000000000001,"15min":2.298},"min":{"1min":0.246,"5min":0.192,"15min":0.192},"max":{"1min":13.428000000000001,"5min":13.459,"15min":16.033999999999999},"last":0.30499999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738992,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6548048,"kb_used_data":5173380,"kb_used_omap":4927,"kb_used_meta":1369728,"kb_avail":87823792,"statfs":{"total":96636764160,"available":89931563008,"internally_reserved":0,"allocated":5297541120,"data_stored":57889085513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1402601933},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,9,6,12,9,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.266,"5min":3.218,"15min":3.1280000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.3320000000000007,"5min":30.896999999999998,"15min":30.896999999999998},"last":5.593},{"interface":"front","average":{"1min":2.3980000000000001,"5min":2.7639999999999998,"15min":2.831},"min":{"1min":0.24299999999999999,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":11.779,"5min":24.917000000000002,"15min":24.917000000000002},"last":5.6109999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":2.4329999999999998,"15min":2.1040000000000001},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":17.084,"5min":17.084,"15min":17.084},"last":4.9400000000000004},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.556,"15min":2.7519999999999998},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":11.736000000000001,"5min":24.867999999999999,"15min":24.867999999999999},"last":4.6790000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3261140992,"data_stored":37983366345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5292888064,"data_stored":57904874820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2031255552,"data_stored":19894286459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:27:07.143 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:07.143 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:07.143 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:07.144 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:07.144 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:27:10.740 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:27:10.738+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:27:10.740 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:27:10.738+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:27:10.741 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:27:10.738+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:27:10.741 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:27:10.738+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:27:10.873 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:10.871+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:11.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.011+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:11.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.011+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:11.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.012+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:11.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.012+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:11.288 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.280+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:11.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.350+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:11.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.350+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:11.402 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.400+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:11.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.757+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:11.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.857+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:11.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.941+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:11.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.985+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:11.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:11.985+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:12.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.016+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:12.021 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.018+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:12.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.117+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:12.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.253+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:12.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.253+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:12.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.358+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:12.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.358+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:12.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.358+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:12.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.365+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:12.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.371+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:12.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.371+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:12.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.402+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:12.404 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.402+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:12.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:12.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:12.451 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:12.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:12.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:12.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:12.447+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:13.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.242+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:13.265 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.258+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:13.365 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.361+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:13.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.364+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:13.382 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.380+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:13.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:13.938+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:14.136 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.134+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:14.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.190+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:14.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.190+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:14.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.228+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:14.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.231+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:14.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:14.231+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:17.734 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:27:17.732+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:27:17.734 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:27:17.732+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:27:20.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.718+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:20.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.718+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:20.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.834+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:20.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.838+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:20.840 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.838+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:20.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.938+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:20.948 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:20.946+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:21.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.051+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:21.064 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.057+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:21.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.067+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:21.068 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.067+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:21.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.571+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:21.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.617+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:21.644 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.641+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:21.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.680+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:21.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.680+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:21.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.686+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:21.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.755+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:21.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:21.766+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:22.355 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.351+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:22.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.356+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:22.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.363+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:22.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.363+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:22.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:22.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:22.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:22.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:22.420 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:22.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:22.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.413+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:22.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.518+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:22.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.577+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:22.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.636+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:22.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.636+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:22.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.663+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:22.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.739+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:22.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.746+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:22.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.746+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:22.761 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.759+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:22.814 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.813+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:22.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.853+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:22.907 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.906+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:22.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.906+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:22.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.906+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:22.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:22.940+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:23.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.008+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:23.097 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.094+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:23.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.113+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:23.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.113+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:23.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.160+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:23.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.223+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:23.228 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:23.225+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:24.626 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:27:24.624+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:27:24.631 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:27:24.630+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:27:27.145 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:27:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:27:27.812 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:27:27.838 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2014,"stamp":"2023-10-03T18:27:27.558868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23810238257,"num_objects":66848,"num_object_clones":61582,"num_object_copies":133696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66848,"num_whiteouts":0,"num_read":1437180,"num_read_kb":5206849,"num_write":1301681,"num_write_kb":5189827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80544,"ondisk_log_size":80544,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15263992,"kb_used_data":10381588,"kb_used_omap":8419,"kb_used_meta":4873948,"kb_avail":267851528,"statfs":{"total":289910292480,"available":274279964672,"internally_reserved":0,"allocated":10630746112,"data_stored":118921084339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8622000,"internal_metadata":4990922832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,6,1,6,4,13,15,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5799936,"num_objects":294,"num_object_clones":294,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":294,"num_whiteouts":0,"num_read":1412,"num_read_kb":5624,"num_write":1410,"num_write_kb":5640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.178511"},"pg_stats":[{"pgid":"2.7","version":"380'195793","reported_seq":345313,"reported_epoch":380,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:22.216577+0000","last_change":"2023-10-03T18:27:21.202366+0000","last_active":"2023-10-03T18:27:22.216577+0000","last_peered":"2023-10-03T18:27:22.216577+0000","last_clean":"2023-10-03T18:27:22.216577+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:27:22.216577+0000","last_undegraded":"2023-10-03T18:27:22.216577+0000","last_fullsized":"2023-10-03T18:27:22.216577+0000","mapping_epoch":12,"log_start":"332'185767","ondisk_log_start":"332'185767","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10026,"log_dups_size":0,"ondisk_log_size":10026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":295,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:03:07.896716+0000","scrub_duration":505.38444223300002,"objects_trimmed":51,"snaptrim_duration":8.5631516009999995,"stat_sum":{"num_bytes":2965774820,"num_objects":11905,"num_object_clones":11254,"num_object_copies":23810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11905,"num_whiteouts":0,"num_read":175092,"num_read_kb":647306,"num_write":161662,"num_write_kb":644765,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"189","length":"2"}]},{"pgid":"2.6","version":"380'181231","reported_seq":320292,"reported_epoch":380,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:22.248127+0000","last_change":"2023-10-03T18:27:21.803817+0000","last_active":"2023-10-03T18:27:22.248127+0000","last_peered":"2023-10-03T18:27:22.248127+0000","last_clean":"2023-10-03T18:27:22.248127+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:27:22.248127+0000","last_undegraded":"2023-10-03T18:27:22.248127+0000","last_fullsized":"2023-10-03T18:27:22.248127+0000","mapping_epoch":12,"log_start":"329'171133","ondisk_log_start":"329'171133","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":210,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":50,"snaptrim_duration":9.2371345369999993,"stat_sum":{"num_bytes":2740335264,"num_objects":8611,"num_object_clones":8006,"num_object_copies":17222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8611,"num_whiteouts":0,"num_read":162543,"num_read_kb":597615,"num_write":149388,"num_write_kb":596280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"189","length":"7"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"}]},{"pgid":"2.5","version":"380'217492","reported_seq":381472,"reported_epoch":380,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:22.242163+0000","last_change":"2023-10-03T18:27:19.421750+0000","last_active":"2023-10-03T18:27:22.242163+0000","last_peered":"2023-10-03T18:27:22.242163+0000","last_clean":"2023-10-03T18:27:22.242163+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:27:22.242163+0000","last_undegraded":"2023-10-03T18:27:22.242163+0000","last_fullsized":"2023-10-03T18:27:22.242163+0000","mapping_epoch":12,"log_start":"334'207438","ondisk_log_start":"334'207438","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10054,"log_dups_size":0,"ondisk_log_size":10054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":241,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":42,"snaptrim_duration":6.8550722300000002,"stat_sum":{"num_bytes":3214287503,"num_objects":11118,"num_object_clones":10408,"num_object_copies":22236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11118,"num_whiteouts":0,"num_read":200331,"num_read_kb":707607,"num_write":176127,"num_write_kb":701120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"189","length":"7"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"1"}]},{"pgid":"2.0","version":"381'211843","reported_seq":366733,"reported_epoch":381,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:24.298932+0000","last_change":"2023-10-03T18:27:23.393375+0000","last_active":"2023-10-03T18:27:24.298932+0000","last_peered":"2023-10-03T18:27:24.298932+0000","last_clean":"2023-10-03T18:27:24.298932+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:27:24.298932+0000","last_undegraded":"2023-10-03T18:27:24.298932+0000","last_fullsized":"2023-10-03T18:27:24.298932+0000","mapping_epoch":12,"log_start":"347'201787","ondisk_log_start":"347'201787","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10056,"log_dups_size":0,"ondisk_log_size":10056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":196,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:46:58.154037+0000","scrub_duration":734.24538164700004,"objects_trimmed":27,"snaptrim_duration":0.77643404299999996,"stat_sum":{"num_bytes":3105202832,"num_objects":9303,"num_object_clones":8618,"num_object_copies":18606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9303,"num_whiteouts":0,"num_read":182320,"num_read_kb":675274,"num_write":169889,"num_write_kb":677675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18a","length":"6"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"4"}]},{"pgid":"1.0","version":"88'99","reported_seq":544,"reported_epoch":379,"state":"active+clean","last_fresh":"2023-10-03T18:27:07.827063+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:27:07.827063+0000","last_peered":"2023-10-03T18:27:07.827063+0000","last_clean":"2023-10-03T18:27:07.827063+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:27:07.827063+0000","last_undegraded":"2023-10-03T18:27:07.827063+0000","last_fullsized":"2023-10-03T18:27:07.827063+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"380'209311","reported_seq":366745,"reported_epoch":380,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:21.995633+0000","last_change":"2023-10-03T18:27:21.832474+0000","last_active":"2023-10-03T18:27:21.995633+0000","last_peered":"2023-10-03T18:27:21.995633+0000","last_clean":"2023-10-03T18:27:21.995633+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:27:21.995633+0000","last_undegraded":"2023-10-03T18:27:21.995633+0000","last_fullsized":"2023-10-03T18:27:21.995633+0000","mapping_epoch":12,"log_start":"332'199242","ondisk_log_start":"332'199242","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10069,"log_dups_size":0,"ondisk_log_size":10069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":232,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":31,"snaptrim_duration":5.2301687159999997,"stat_sum":{"num_bytes":3158139671,"num_objects":10622,"num_object_clones":9924,"num_object_copies":21244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10622,"num_whiteouts":0,"num_read":185583,"num_read_kb":686814,"num_write":172256,"num_write_kb":687073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"189","length":"7"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"8"}]},{"pgid":"2.1","version":"381'205754","reported_seq":348651,"reported_epoch":381,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:24.396789+0000","last_change":"2023-10-03T18:27:22.537563+0000","last_active":"2023-10-03T18:27:24.396789+0000","last_peered":"2023-10-03T18:27:24.396789+0000","last_clean":"2023-10-03T18:27:24.396789+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:27:24.396789+0000","last_undegraded":"2023-10-03T18:27:24.396789+0000","last_fullsized":"2023-10-03T18:27:24.396789+0000","mapping_epoch":12,"log_start":"337'195735","ondisk_log_start":"337'195735","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:49.170346+0000","scrub_duration":424.08306342600002,"objects_trimmed":26,"snaptrim_duration":9.9644594879999993,"stat_sum":{"num_bytes":2877854420,"num_objects":3046,"num_object_clones":2409,"num_object_copies":6092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":175096,"num_read_kb":628218,"num_write":157300,"num_write_kb":626667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18a","length":"6"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"}]},{"pgid":"2.2","version":"381'205930","reported_seq":354092,"reported_epoch":381,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:25.793526+0000","last_change":"2023-10-03T18:27:25.169734+0000","last_active":"2023-10-03T18:27:25.793526+0000","last_peered":"2023-10-03T18:27:25.793526+0000","last_clean":"2023-10-03T18:27:25.793526+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:27:25.793526+0000","last_undegraded":"2023-10-03T18:27:25.793526+0000","last_fullsized":"2023-10-03T18:27:25.793526+0000","mapping_epoch":12,"log_start":"336'195883","ondisk_log_start":"336'195883","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:58:54.187056+0000","scrub_duration":370.574046178,"objects_trimmed":55,"snaptrim_duration":2.5894957569999999,"stat_sum":{"num_bytes":2881151750,"num_objects":3082,"num_object_clones":2440,"num_object_copies":6164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":184862,"num_read_kb":639727,"num_write":158622,"num_write_kb":631069,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"18a","length":"6"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"}]},{"pgid":"2.4","version":"380'191306","reported_seq":336459,"reported_epoch":380,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:22.020865+0000","last_change":"2023-10-03T18:27:21.222850+0000","last_active":"2023-10-03T18:27:22.020865+0000","last_peered":"2023-10-03T18:27:22.020865+0000","last_clean":"2023-10-03T18:27:22.020865+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:27:22.020865+0000","last_undegraded":"2023-10-03T18:27:22.020865+0000","last_fullsized":"2023-10-03T18:27:22.020865+0000","mapping_epoch":12,"log_start":"331'181230","ondisk_log_start":"331'181230","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":217,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":2.9402749749999999,"stat_sum":{"num_bytes":2866901629,"num_objects":9159,"num_object_clones":8523,"num_object_copies":18318,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9159,"num_whiteouts":0,"num_read":171247,"num_read_kb":624197,"num_write":156300,"num_write_kb":623284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"189","length":"7"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23809647889,"num_objects":66846,"num_object_clones":61582,"num_object_copies":133692,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66846,"num_whiteouts":0,"num_read":1437074,"num_read_kb":5206758,"num_write":1301544,"num_write_kb":5187933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10618413056,"data_stored":118998932002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":68668,"internal_metadata":0},"log_size":80445,"ondisk_log_size":80445,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739024,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3236184,"kb_used_data":1994508,"kb_used_omap":1375,"kb_used_meta":1240288,"kb_avail":91135656,"statfs":{"total":96636764160,"available":93322911744,"internally_reserved":0,"allocated":2042376192,"data_stored":20383754790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1270055044},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,0,4,4,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":2.411,"5min":2.46,"15min":2.5470000000000002},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.922000000000001,"5min":15.964,"15min":18.321000000000002},"last":1.0609999999999999},{"interface":"front","average":{"1min":4.2839999999999998,"5min":2.653,"15min":2.4510000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20100000000000001},"max":{"1min":34.176000000000002,"5min":34.176000000000002,"15min":34.176000000000002},"last":1.004}]},{"osd":1,"last update":"Tue Oct 3 18:26:24 2023","interfaces":[{"interface":"back","average":{"1min":4.2110000000000003,"5min":2.609,"15min":2.508},"min":{"1min":0.27900000000000003,"5min":0.25,"15min":0.221},"max":{"1min":15.85,"5min":15.85,"15min":18.768999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":3.8519999999999999,"5min":2.661,"15min":2.4079999999999999},"min":{"1min":0.215,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":13.842000000000001,"5min":13.842000000000001,"15min":18.751999999999999},"last":0.38300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739023,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5325820,"kb_used_data":3199988,"kb_used_omap":2116,"kb_used_meta":2123707,"kb_avail":89046020,"statfs":{"total":96636764160,"available":91183124480,"internally_reserved":0,"allocated":3276787712,"data_stored":39154102369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2167297,"internal_metadata":2174676479},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:26:43 2023","interfaces":[{"interface":"back","average":{"1min":2.5430000000000001,"5min":3.1960000000000002,"15min":3.1869999999999998},"min":{"1min":0.188,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":13.396000000000001,"5min":45.886000000000003,"15min":45.886000000000003},"last":4.9489999999999998},{"interface":"front","average":{"1min":2.617,"5min":3.0960000000000001,"15min":3.3069999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":13.538,"5min":45.555,"15min":45.555},"last":8.3149999999999995}]},{"osd":2,"last update":"Tue Oct 3 18:26:43 2023","interfaces":[{"interface":"back","average":{"1min":2.4860000000000002,"5min":2.0859999999999999,"15min":1.978},"min":{"1min":0.221,"5min":0.20899999999999999,"15min":0.192},"max":{"1min":11.252000000000001,"5min":12.752000000000001,"15min":13.209},"last":0.66600000000000004},{"interface":"front","average":{"1min":3.5670000000000002,"5min":2.4390000000000001,"15min":2.298},"min":{"1min":0.246,"5min":0.192,"15min":0.192},"max":{"1min":13.428000000000001,"5min":13.459,"15min":16.033999999999999},"last":2.5049999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738995,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6701988,"kb_used_data":5187092,"kb_used_omap":4927,"kb_used_meta":1509952,"kb_avail":87669852,"statfs":{"total":96636764160,"available":89773928448,"internally_reserved":0,"allocated":5311582208,"data_stored":59383227180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1546191309},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,0,5,2,8,11,9,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.266,"5min":3.218,"15min":3.1280000000000001},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.3320000000000007,"5min":30.896999999999998,"15min":30.896999999999998},"last":5.9820000000000002},{"interface":"front","average":{"1min":2.3980000000000001,"5min":2.7639999999999998,"15min":2.831},"min":{"1min":0.24299999999999999,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":11.779,"5min":24.917000000000002,"15min":24.917000000000002},"last":5.415}]},{"osd":2,"last update":"Tue Oct 3 18:26:32 2023","interfaces":[{"interface":"back","average":{"1min":2.782,"5min":2.4329999999999998,"15min":2.1040000000000001},"min":{"1min":0.216,"5min":0.20899999999999999,"15min":0.20799999999999999},"max":{"1min":17.084,"5min":17.084,"15min":17.084},"last":5.3319999999999999},{"interface":"front","average":{"1min":1.7030000000000001,"5min":2.556,"15min":2.7519999999999998},"min":{"1min":0.215,"5min":0.186,"15min":0.186},"max":{"1min":11.736000000000001,"5min":24.867999999999999,"15min":24.867999999999999},"last":5.452}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3272450048,"data_stored":39168441494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":26270,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5307768832,"data_stored":59443893521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2038194176,"data_stored":20386596987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:27:27.840 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:27.840 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:27.840 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:27.840 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:27.841 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:27:28.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:28.736+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:28.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:28.736+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:28.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:28.948+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:29.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:29.041+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:29.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:29.041+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:29.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:29.047+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:29.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:29.239+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:29.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:29.245+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:31.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.463+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:31.465 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.463+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:31.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.470+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:31.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.470+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:31.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.491+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:31.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.587+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:31.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.587+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:31.625 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.622+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:31.692 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.692+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:31.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:31.802+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:33.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.473+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:27:33.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.473+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:33.474 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.473+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:33.578 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.577+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.577+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:33.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:33.578+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:34.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.504+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:34.506 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.504+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:34.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.606+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:34.690 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.684+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:34.718 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.718+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:34.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.718+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:34.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.718+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:34.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.718+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:34.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.769+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:34.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:34.793+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:37.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.584+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:37.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:37.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.584+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:37.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.593+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:37.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.670+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:37.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.670+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:37.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.688+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:37.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:37.710+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:39.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.883+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:27:39.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.946+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:39.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.946+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:39.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.949+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:39.967 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.966+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:39.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.966+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:39.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.966+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:39.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.966+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:39.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:39.966+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:40.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:40.225+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:40.226 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:40.225+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:40.257 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:40.254+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:42.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.754+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:27:42.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.762+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:42.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.788+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:42.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.788+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:42.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.788+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:42.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.788+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:42.903 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.897+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:42.903 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:42.897+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:47.337 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.325+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:47.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.355+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:47.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.424+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:47.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.432+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:47.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.432+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:47.436 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.432+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:47.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.521+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:47.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:47.521+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:47.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:27:48.729 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:27:48.749 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:27:48.768 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2027,"stamp":"2023-10-03T18:27:47.557720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23834335025,"num_objects":66952,"num_object_clones":61686,"num_object_copies":133904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66952,"num_whiteouts":0,"num_read":1444336,"num_read_kb":5231733,"num_write":1307508,"num_write_kb":5213135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80545,"ondisk_log_size":80545,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14214036,"kb_used_data":10428344,"kb_used_omap":9077,"kb_used_meta":3776586,"kb_avail":268901484,"statfs":{"total":289910292480,"available":275355119616,"internally_reserved":0,"allocated":10678624256,"data_stored":119392274067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":3867224240},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,2,7,4,12,10,13],"upper_bound":256},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8724480,"num_objects":-210,"num_object_clones":-210,"num_object_copies":-420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-210,"num_whiteouts":0,"num_read":3316,"num_read_kb":9588,"num_write":2117,"num_write_kb":8468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.168723"},"pg_stats":[{"pgid":"2.7","version":"384'196963","reported_seq":347045,"reported_epoch":384,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:43.886683+0000","last_change":"2023-10-03T18:27:42.797212+0000","last_active":"2023-10-03T18:27:43.886683+0000","last_peered":"2023-10-03T18:27:43.886683+0000","last_clean":"2023-10-03T18:27:43.886683+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:27:43.886683+0000","last_undegraded":"2023-10-03T18:27:43.886683+0000","last_fullsized":"2023-10-03T18:27:43.886683+0000","mapping_epoch":12,"log_start":"336'186867","ondisk_log_start":"336'186867","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":292,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:03:07.896716+0000","scrub_duration":505.38444223300002,"objects_trimmed":16,"snaptrim_duration":6.2400662799999997,"stat_sum":{"num_bytes":2968859108,"num_objects":11988,"num_object_clones":11337,"num_object_copies":23976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11988,"num_whiteouts":0,"num_read":175972,"num_read_kb":650444,"num_write":162406,"num_write_kb":647741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18d","length":"1"}]},{"pgid":"2.6","version":"384'182282","reported_seq":321877,"reported_epoch":384,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:43.901467+0000","last_change":"2023-10-03T18:27:40.822008+0000","last_active":"2023-10-03T18:27:43.901467+0000","last_peered":"2023-10-03T18:27:43.901467+0000","last_clean":"2023-10-03T18:27:43.901467+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:27:43.901467+0000","last_undegraded":"2023-10-03T18:27:43.901467+0000","last_fullsized":"2023-10-03T18:27:43.901467+0000","mapping_epoch":12,"log_start":"332'172236","ondisk_log_start":"332'172236","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":207,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":8.2496506370000002,"stat_sum":{"num_bytes":2743055008,"num_objects":8696,"num_object_clones":8091,"num_object_copies":17392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8696,"num_whiteouts":0,"num_read":163337,"num_read_kb":600553,"num_write":150046,"num_write_kb":598912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"3"}]},{"pgid":"2.5","version":"384'218845","reported_seq":383493,"reported_epoch":384,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:43.917920+0000","last_change":"2023-10-03T18:27:42.947579+0000","last_active":"2023-10-03T18:27:43.917920+0000","last_peered":"2023-10-03T18:27:43.917920+0000","last_clean":"2023-10-03T18:27:43.917920+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:27:43.917920+0000","last_undegraded":"2023-10-03T18:27:43.917920+0000","last_fullsized":"2023-10-03T18:27:43.917920+0000","mapping_epoch":12,"log_start":"338'208744","ondisk_log_start":"338'208744","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10101,"log_dups_size":0,"ondisk_log_size":10101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":238,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":16,"snaptrim_duration":8.1038035639999997,"stat_sum":{"num_bytes":3217818255,"num_objects":11213,"num_object_clones":10503,"num_object_copies":22426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11213,"num_whiteouts":0,"num_read":201391,"num_read_kb":711264,"num_write":176987,"num_write_kb":704560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"2"}]},{"pgid":"2.0","version":"384'213341","reported_seq":368689,"reported_epoch":384,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:27:44.646705+0000","last_change":"2023-10-03T18:27:42.971868+0000","last_active":"2023-10-03T18:27:44.646705+0000","last_peered":"2023-10-03T18:27:44.646705+0000","last_clean":"2023-10-03T18:27:44.646705+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:27:44.646705+0000","last_undegraded":"2023-10-03T18:27:44.646705+0000","last_fullsized":"2023-10-03T18:27:44.646705+0000","mapping_epoch":12,"log_start":"349'203299","ondisk_log_start":"349'203299","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":187,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:46:58.154037+0000","scrub_duration":734.24538164700004,"objects_trimmed":32,"snaptrim_duration":0.361773234,"stat_sum":{"num_bytes":3108180624,"num_objects":9186,"num_object_clones":8501,"num_object_copies":18372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9186,"num_whiteouts":0,"num_read":183153,"num_read_kb":678216,"num_write":170619,"num_write_kb":680595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":548,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-10-03T18:27:32.659058+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:27:32.659058+0000","last_peered":"2023-10-03T18:27:32.659058+0000","last_clean":"2023-10-03T18:27:32.659058+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:27:32.659058+0000","last_undegraded":"2023-10-03T18:27:32.659058+0000","last_fullsized":"2023-10-03T18:27:32.659058+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"384'210508","reported_seq":368574,"reported_epoch":384,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:43.917946+0000","last_change":"2023-10-03T18:27:42.947092+0000","last_active":"2023-10-03T18:27:43.917946+0000","last_peered":"2023-10-03T18:27:43.917946+0000","last_clean":"2023-10-03T18:27:43.917946+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:27:43.917946+0000","last_undegraded":"2023-10-03T18:27:43.917946+0000","last_fullsized":"2023-10-03T18:27:43.917946+0000","mapping_epoch":12,"log_start":"336'200446","ondisk_log_start":"336'200446","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":228,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.30707994,"stat_sum":{"num_bytes":3161187095,"num_objects":10709,"num_object_clones":10011,"num_object_copies":21418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10709,"num_whiteouts":0,"num_read":186536,"num_read_kb":690224,"num_write":172993,"num_write_kb":690021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"2"}]},{"pgid":"2.1","version":"384'207156","reported_seq":350506,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-10-03T18:27:44.631289+0000","last_change":"2023-10-03T18:27:41.879050+0000","last_active":"2023-10-03T18:27:44.631289+0000","last_peered":"2023-10-03T18:27:44.631289+0000","last_clean":"2023-10-03T18:27:44.631289+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:27:44.631289+0000","last_undegraded":"2023-10-03T18:27:44.631289+0000","last_fullsized":"2023-10-03T18:27:44.631289+0000","mapping_epoch":12,"log_start":"347'197142","ondisk_log_start":"347'197142","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:36:49.170346+0000","scrub_duration":424.08306342600002,"objects_trimmed":150,"snaptrim_duration":9.3135467300000006,"stat_sum":{"num_bytes":2880746196,"num_objects":2942,"num_object_clones":2305,"num_object_copies":5884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":175937,"num_read_kb":631003,"num_write":157993,"num_write_kb":629439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"}]},{"pgid":"2.2","version":"384'207336","reported_seq":356018,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-10-03T18:27:45.992435+0000","last_change":"2023-10-03T18:27:44.866239+0000","last_active":"2023-10-03T18:27:45.992435+0000","last_peered":"2023-10-03T18:27:45.992435+0000","last_clean":"2023-10-03T18:27:45.992435+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:27:45.992435+0000","last_undegraded":"2023-10-03T18:27:45.992435+0000","last_fullsized":"2023-10-03T18:27:45.992435+0000","mapping_epoch":12,"log_start":"347'197290","ondisk_log_start":"347'197290","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:58:54.187056+0000","scrub_duration":370.574046178,"objects_trimmed":35,"snaptrim_duration":2.2774677570000001,"stat_sum":{"num_bytes":2883949318,"num_objects":2962,"num_object_clones":2320,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":185838,"num_read_kb":642718,"num_write":159299,"num_write_kb":633777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"}]},{"pgid":"2.4","version":"384'192472","reported_seq":338180,"reported_epoch":384,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:27:43.908208+0000","last_change":"2023-10-03T18:27:42.791026+0000","last_active":"2023-10-03T18:27:43.908208+0000","last_peered":"2023-10-03T18:27:43.908208+0000","last_clean":"2023-10-03T18:27:43.908208+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:27:43.908208+0000","last_undegraded":"2023-10-03T18:27:43.908208+0000","last_fullsized":"2023-10-03T18:27:43.908208+0000","mapping_epoch":12,"log_start":"334'182433","ondisk_log_start":"334'182433","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":213,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.194814446,"stat_sum":{"num_bytes":2869949053,"num_objects":9254,"num_object_clones":8618,"num_object_copies":18508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9254,"num_whiteouts":0,"num_read":172066,"num_read_kb":627220,"num_write":157028,"num_write_kb":626196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"18d","length":"3"},{"start":"199","length":"5"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"7"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23833744657,"num_objects":66950,"num_object_clones":61686,"num_object_copies":133900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66950,"num_whiteouts":0,"num_read":1444230,"num_read_kb":5231642,"num_write":1307371,"num_write_kb":5211241,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10665967616,"data_stored":119377967650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80446,"ondisk_log_size":80446,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739028,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3285576,"kb_used_data":2002944,"kb_used_omap":1375,"kb_used_meta":1281248,"kb_avail":91086264,"statfs":{"total":96636764160,"available":93272334336,"internally_reserved":0,"allocated":2051014656,"data_stored":20317498166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1311998084},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,0,4,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5569999999999999,"5min":2.3319999999999999,"15min":2.5880000000000001},"min":{"1min":0.19700000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":11.4,"5min":12.737,"15min":18.321000000000002},"last":2.859},{"interface":"front","average":{"1min":2.2010000000000001,"5min":2.6930000000000001,"15min":2.4590000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":9.6259999999999994,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.32300000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":2.6989999999999998,"15min":2.5390000000000001},"min":{"1min":0.255,"5min":0.25,"15min":0.221},"max":{"1min":9.5920000000000005,"5min":15.85,"15min":18.768999999999998},"last":6.3689999999999998},{"interface":"front","average":{"1min":1.849,"5min":2.423,"15min":2.3479999999999999},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":8.3740000000000006,"5min":13.842000000000001,"15min":18.751999999999999},"last":5.7080000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739027,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4143388,"kb_used_data":3214164,"kb_used_omap":2774,"kb_used_meta":926441,"kb_avail":90228452,"statfs":{"total":96636764160,"available":92393934848,"internally_reserved":0,"allocated":3291303936,"data_stored":39376613745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":948676191},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,3,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":4.8979999999999997,"5min":3.758,"15min":3.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":21.256,"5min":45.886000000000003,"15min":45.886000000000003},"last":10.282999999999999},{"interface":"front","average":{"1min":3.8519999999999999,"5min":3.29,"15min":3.2770000000000001},"min":{"1min":0.20399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":19.018999999999998,"5min":45.555,"15min":45.555},"last":0.70099999999999996}]},{"osd":2,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.113,"15min":2.0049999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":8.9860000000000007,"5min":11.252000000000001,"15min":13.209},"last":8.9860000000000007},{"interface":"front","average":{"1min":5.4249999999999998,"5min":3,"15min":2.5230000000000001},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":25.77,"5min":25.77,"15min":25.77},"last":2.2360000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738999,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6785072,"kb_used_data":5211236,"kb_used_omap":4927,"kb_used_meta":1568896,"kb_avail":87586768,"statfs":{"total":96636764160,"available":89688850432,"internally_reserved":0,"allocated":5336305664,"data_stored":59698162156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1606549965},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,5,1,5,9,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.3170000000000002,"5min":3.3799999999999999,"15min":3.274},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":20.466999999999999,"5min":22.068999999999999,"15min":30.896999999999998},"last":2.4700000000000002},{"interface":"front","average":{"1min":5.0140000000000002,"5min":3.2589999999999999,"15min":3.044},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":20.436,"5min":24.917000000000002,"15min":24.917000000000002},"last":2.3069999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.21,"5min":2.8860000000000001,"15min":2.3069999999999999},"min":{"1min":0.34000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":34.395000000000003,"5min":34.395000000000003,"15min":34.395000000000003},"last":2.8919999999999999},{"interface":"front","average":{"1min":5.5259999999999998,"5min":3.153,"15min":2.9870000000000001},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":36.332999999999998,"5min":36.332999999999998,"15min":36.332999999999998},"last":2.9319999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3287117824,"data_stored":39367945366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5331959808,"data_stored":59694318865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2046889984,"data_stored":20315703419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:27:48.770 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:48.770 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:48.771 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:48.771 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:27:48.771 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:27:49.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.091+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:49.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.103+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:49.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.103+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:49.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.103+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:49.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.122+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:49.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:49.125+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:52.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.137+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:52.144 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.143+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:52.144 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.143+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:52.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.151+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:52.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.151+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:52.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:52.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:54.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.796+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:27:54.816 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.815+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:54.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.876+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:54.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.876+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:54.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.876+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:54.972 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.968+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:27:54.973 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:54.968+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:27:59.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.401+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:27:59.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.428+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:27:59.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.446+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:27:59.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.446+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:27:59.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.446+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:27:59.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:27:59.495+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:00.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.279+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:00.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.279+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:00.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.279+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:00.301 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.301+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:00.302 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.301+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:00.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.362+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:00.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.362+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:00.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:00.400+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:01.912 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:01.906+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:01.925 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:01.923+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:01.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:01.923+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:02.004 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.002+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:02.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.152+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:02.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.152+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:02.221 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.211+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:02.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.227+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:02.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.349+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:02.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.349+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:02.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.349+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:02.407 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:28:02.405+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:28:02.407 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:28:02.405+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:28:02.408 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:28:02.407+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:28:02.408 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:28:02.407+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:28:02.505 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.499+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:02.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.550+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:02.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:02.550+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:03.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:03.947+0000 7fc515f98700 -1 reset not still connected to 0x5622282760d0 2023-10-03T18:28:04.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:04.088+0000 7fc515f98700 -1 reset not still connected to 0x56222f80e9c0 2023-10-03T18:28:05.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:05.750+0000 7fc515f98700 -1 reset not still connected to 0x5622135dde10 2023-10-03T18:28:06.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:05.750+0000 7fc515f98700 -1 reset not still connected to 0x5622179baf70 2023-10-03T18:28:06.022 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:05.750+0000 7fc515f98700 -1 reset not still connected to 0x56222f80e9c0 2023-10-03T18:28:06.237 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.235+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:06.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.289+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:06.429 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.289+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:06.430 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.397+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:06.575 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.573+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:06.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.613+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:06.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.613+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:06.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.613+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:06.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.613+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:06.752 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.659+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:06.777 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.774+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:06.978 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.977+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:06.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:06.994+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:07.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.144+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:07.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.184+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:07.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:07.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:07.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:07.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:07.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:07.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:07.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:07.194+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:08.772 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:28:09.432 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:28:09.464 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:28:09.473 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2048,"stamp":"2023-10-03T18:28:09.398077+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23863797655,"num_objects":66979,"num_object_clones":61711,"num_object_copies":133958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66979,"num_whiteouts":0,"num_read":1454665,"num_read_kb":5263012,"num_write":1314777,"num_write_kb":5241591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80492,"ondisk_log_size":80492,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14468104,"kb_used_data":10484648,"kb_used_omap":9077,"kb_used_meta":3974346,"kb_avail":268647416,"statfs":{"total":289910292480,"available":275094953984,"internally_reserved":0,"allocated":10736279552,"data_stored":119624667073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":4069730480},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,1,0,1,6,8,17,18,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":42,"apply_latency_ms":42,"commit_latency_ns":42000000,"apply_latency_ns":42000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5853236,"num_objects":293,"num_object_clones":292,"num_object_copies":586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":293,"num_whiteouts":0,"num_read":4003,"num_read_kb":7878,"num_write":1583,"num_write_kb":5751,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.790623"},"pg_stats":[{"pgid":"2.7","version":"392'198297","reported_seq":349300,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:05.677413+0000","last_change":"2023-10-03T18:28:05.574450+0000","last_active":"2023-10-03T18:28:05.677413+0000","last_peered":"2023-10-03T18:28:05.677413+0000","last_clean":"2023-10-03T18:28:05.677413+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:28:05.677413+0000","last_undegraded":"2023-10-03T18:28:05.677413+0000","last_fullsized":"2023-10-03T18:28:05.677413+0000","mapping_epoch":12,"log_start":"346'188269","ondisk_log_start":"346'188269","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":301,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:57:27.068604+0000","scrub_duration":505.38444223300002,"objects_trimmed":4,"snaptrim_duration":0.92080387399999997,"stat_sum":{"num_bytes":2972533221,"num_objects":11984,"num_object_clones":11333,"num_object_copies":23968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11984,"num_whiteouts":0,"num_read":177284,"num_read_kb":654732,"num_write":163303,"num_write_kb":651268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19b","length":"1"}]},{"pgid":"2.6","version":"392'183461","reported_seq":323869,"reported_epoch":392,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:05.611193+0000","last_change":"2023-10-03T18:28:05.610942+0000","last_active":"2023-10-03T18:28:05.611193+0000","last_peered":"2023-10-03T18:28:05.611193+0000","last_clean":"2023-10-03T18:28:05.611193+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:28:05.611193+0000","last_undegraded":"2023-10-03T18:28:05.611193+0000","last_fullsized":"2023-10-03T18:28:05.611193+0000","mapping_epoch":12,"log_start":"336'173440","ondisk_log_start":"336'173440","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":217,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":2.6986138,"stat_sum":{"num_bytes":2746540604,"num_objects":8710,"num_object_clones":8107,"num_object_copies":17420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8710,"num_whiteouts":0,"num_read":164443,"num_read_kb":604022,"num_write":150895,"num_write_kb":602264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"7"}]},{"pgid":"2.5","version":"392'220320","reported_seq":385895,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:05.630173+0000","last_change":"2023-10-03T18:28:05.470567+0000","last_active":"2023-10-03T18:28:05.630173+0000","last_peered":"2023-10-03T18:28:05.630173+0000","last_clean":"2023-10-03T18:28:05.630173+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:28:05.630173+0000","last_undegraded":"2023-10-03T18:28:05.630173+0000","last_fullsized":"2023-10-03T18:28:05.630173+0000","mapping_epoch":12,"log_start":"348'210248","ondisk_log_start":"348'210248","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":248,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.78700990800000004,"stat_sum":{"num_bytes":3222074048,"num_objects":11234,"num_object_clones":10523,"num_object_copies":22468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11234,"num_whiteouts":0,"num_read":202923,"num_read_kb":715569,"num_write":178027,"num_write_kb":708613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"6"}]},{"pgid":"2.0","version":"392'215023","reported_seq":371150,"reported_epoch":392,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:04.777270+0000","last_change":"2023-10-03T18:28:04.226446+0000","last_active":"2023-10-03T18:28:04.777270+0000","last_peered":"2023-10-03T18:28:04.777270+0000","last_clean":"2023-10-03T18:28:04.777270+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:28:04.777270+0000","last_undegraded":"2023-10-03T18:28:04.777270+0000","last_fullsized":"2023-10-03T18:28:04.777270+0000","mapping_epoch":12,"log_start":"353'205010","ondisk_log_start":"353'205010","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":191,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:38:05.523820+0000","scrub_duration":734.24538164700004,"objects_trimmed":2,"snaptrim_duration":0.59937273000000002,"stat_sum":{"num_bytes":3111973571,"num_objects":9018,"num_object_clones":8332,"num_object_copies":18036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9018,"num_whiteouts":0,"num_read":184370,"num_read_kb":682144,"num_write":171567,"num_write_kb":684326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"3"}]},{"pgid":"1.0","version":"88'99","reported_seq":557,"reported_epoch":391,"state":"active+clean","last_fresh":"2023-10-03T18:28:03.675704+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:28:03.675704+0000","last_peered":"2023-10-03T18:28:03.675704+0000","last_clean":"2023-10-03T18:28:03.675704+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:28:03.675704+0000","last_undegraded":"2023-10-03T18:28:03.675704+0000","last_fullsized":"2023-10-03T18:28:03.675704+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"392'211796","reported_seq":370733,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:05.441374+0000","last_change":"2023-10-03T18:28:00.065032+0000","last_active":"2023-10-03T18:28:05.441374+0000","last_peered":"2023-10-03T18:28:05.441374+0000","last_clean":"2023-10-03T18:28:05.441374+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:28:05.441374+0000","last_undegraded":"2023-10-03T18:28:05.441374+0000","last_fullsized":"2023-10-03T18:28:05.441374+0000","mapping_epoch":12,"log_start":"346'201751","ondisk_log_start":"346'201751","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":239,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.4895128180000001,"stat_sum":{"num_bytes":3165012759,"num_objects":10741,"num_object_clones":10043,"num_object_copies":21482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10741,"num_whiteouts":0,"num_read":187786,"num_read_kb":694258,"num_write":173924,"num_write_kb":693673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"5"}]},{"pgid":"2.1","version":"392'208229","reported_seq":352430,"reported_epoch":392,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:04.757868+0000","last_change":"2023-10-03T18:28:02.754144+0000","last_active":"2023-10-03T18:28:04.757868+0000","last_peered":"2023-10-03T18:28:04.757868+0000","last_clean":"2023-10-03T18:28:04.757868+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:28:04.757868+0000","last_undegraded":"2023-10-03T18:28:04.757868+0000","last_fullsized":"2023-10-03T18:28:04.757868+0000","mapping_epoch":12,"log_start":"348'198143","ondisk_log_start":"348'198143","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:43:44.712408+0000","scrub_duration":424.08306342600002,"objects_trimmed":0,"snaptrim_duration":0.15767964500000001,"stat_sum":{"num_bytes":2884174549,"num_objects":2987,"num_object_clones":2350,"num_object_copies":5974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2987,"num_whiteouts":0,"num_read":177166,"num_read_kb":634753,"num_write":158846,"num_write_kb":632777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"1"}]},{"pgid":"2.2","version":"393'208424","reported_seq":357948,"reported_epoch":393,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:06.156100+0000","last_change":"2023-10-03T18:28:05.354787+0000","last_active":"2023-10-03T18:28:06.156100+0000","last_peered":"2023-10-03T18:28:06.156100+0000","last_clean":"2023-10-03T18:28:06.156100+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:28:06.156100+0000","last_undegraded":"2023-10-03T18:28:06.156100+0000","last_fullsized":"2023-10-03T18:28:06.156100+0000","mapping_epoch":12,"log_start":"348'198392","ondisk_log_start":"348'198392","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:25:15.676908+0000","scrub_duration":370.574046178,"objects_trimmed":4,"snaptrim_duration":0.69380561399999996,"stat_sum":{"num_bytes":2887279365,"num_objects":3025,"num_object_clones":2383,"num_object_copies":6050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":187233,"num_read_kb":646359,"num_write":160146,"num_write_kb":637035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"2"}]},{"pgid":"2.4","version":"392'193732","reported_seq":340308,"reported_epoch":392,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:05.534853+0000","last_change":"2023-10-03T18:28:02.045477+0000","last_active":"2023-10-03T18:28:05.534853+0000","last_peered":"2023-10-03T18:28:05.534853+0000","last_clean":"2023-10-03T18:28:05.534853+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:28:05.534853+0000","last_undegraded":"2023-10-03T18:28:05.534853+0000","last_fullsized":"2023-10-03T18:28:05.534853+0000","mapping_epoch":12,"log_start":"338'183636","ondisk_log_start":"338'183636","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10096,"log_dups_size":0,"ondisk_log_size":10096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":223,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.37332120499999999,"stat_sum":{"num_bytes":2873619170,"num_objects":9278,"num_object_clones":8640,"num_object_copies":18556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9278,"num_whiteouts":0,"num_read":173354,"num_read_kb":631084,"num_write":157932,"num_write_kb":629741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19b","length":"3"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23863207287,"num_objects":66977,"num_object_clones":61711,"num_object_copies":133954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66977,"num_whiteouts":0,"num_read":1454559,"num_read_kb":5262921,"num_write":1314640,"num_write_kb":5239697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10723278848,"data_stored":119818836616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80393,"ondisk_log_size":80393,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739032,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3323892,"kb_used_data":2013476,"kb_used_omap":1375,"kb_used_meta":1309024,"kb_avail":91047948,"statfs":{"total":96636764160,"available":93233098752,"internally_reserved":0,"allocated":2061799424,"data_stored":20246305042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1340440708},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,0,0,0,2,1,4,2,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5569999999999999,"5min":2.3319999999999999,"15min":2.5880000000000001},"min":{"1min":0.19700000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":11.4,"5min":12.737,"15min":18.321000000000002},"last":0.94699999999999995},{"interface":"front","average":{"1min":2.2010000000000001,"5min":2.6930000000000001,"15min":2.4590000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":9.6259999999999994,"5min":34.176000000000002,"15min":34.176000000000002},"last":8.9459999999999997}]},{"osd":1,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":2.6989999999999998,"15min":2.5390000000000001},"min":{"1min":0.255,"5min":0.25,"15min":0.221},"max":{"1min":9.5920000000000005,"5min":15.85,"15min":18.768999999999998},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.849,"5min":2.423,"15min":2.3479999999999999},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":8.3740000000000006,"5min":13.842000000000001,"15min":18.751999999999999},"last":0.26300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739031,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4155668,"kb_used_data":3231372,"kb_used_omap":2774,"kb_used_meta":921513,"kb_avail":90216172,"statfs":{"total":96636764160,"available":92381360128,"internally_reserved":0,"allocated":3308924928,"data_stored":39636975183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":943629919},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,0,1,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":4.8979999999999997,"5min":3.758,"15min":3.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":21.256,"5min":45.886000000000003,"15min":45.886000000000003},"last":5.7939999999999996},{"interface":"front","average":{"1min":3.8519999999999999,"5min":3.29,"15min":3.2770000000000001},"min":{"1min":0.20399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":19.018999999999998,"5min":45.555,"15min":45.555},"last":1.464}]},{"osd":2,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.113,"15min":2.0049999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":8.9860000000000007,"5min":11.252000000000001,"15min":13.209},"last":0.76900000000000002},{"interface":"front","average":{"1min":5.4249999999999998,"5min":3,"15min":2.5230000000000001},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":25.77,"5min":25.77,"15min":25.77},"last":0.32500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739003,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6988544,"kb_used_data":5239800,"kb_used_omap":4927,"kb_used_meta":1743808,"kb_avail":87383296,"statfs":{"total":96636764160,"available":89480495104,"internally_reserved":0,"allocated":5365555200,"data_stored":59741386848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1785659853},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,7,12,15,8],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.3170000000000002,"5min":3.3799999999999999,"15min":3.274},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":20.466999999999999,"5min":22.068999999999999,"15min":30.896999999999998},"last":0.504},{"interface":"front","average":{"1min":5.0140000000000002,"5min":3.2589999999999999,"15min":3.044},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":20.436,"5min":24.917000000000002,"15min":24.917000000000002},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.21,"5min":2.8860000000000001,"15min":2.3069999999999999},"min":{"1min":0.34000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":34.395000000000003,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":5.5259999999999998,"5min":3.153,"15min":2.9870000000000001},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":36.332999999999998,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.253}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3304005632,"data_stored":39641058504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5361655808,"data_stored":59909575953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2057617408,"data_stored":20268202159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:28:09.475 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:09.475 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:09.476 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:09.476 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:09.476 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:28:11.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:11.896+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:11.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:11.937+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:11.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:11.937+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:12.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.051+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:12.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.092+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:12.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.149+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:12.151 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.149+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:12.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.191+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:12.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.191+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:12.225 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.224+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:12.990 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:12.987+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:13.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.122+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:13.129 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.128+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:13.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.156+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:13.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.163+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:13.164 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.163+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:13.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.531+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:13.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.577+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:13.604 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.601+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:13.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.891+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:13.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.891+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:13.918 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:13.916+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:14.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.072+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:14.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.114+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:14.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.151+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:14.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.151+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:14.231 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.228+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:14.248 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.247+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:14.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:14.366+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:18.953 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:18.952+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:18.954 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:18.952+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:19.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.030+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:19.032 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.030+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:19.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.085+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:19.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.085+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:19.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.085+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:19.127 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.125+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:19.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.170+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:19.627 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.623+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:19.747 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.743+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:19.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.762+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:19.785 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.783+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:19.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.790+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:19.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.852+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:19.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.852+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:19.874 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.865+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:19.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:19.907+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:20.053 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:20.044+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:24.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.491+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:24.613 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.609+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:24.615 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.614+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.655+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.655+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.655+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.655+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:24.657 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.655+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:24.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.657+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:24.658 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.657+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:24.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.801+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:24.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:24.923+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:25.117 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.111+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:25.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.111+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:25.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.114+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:25.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.114+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:25.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.114+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:25.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.150+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:25.152 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.150+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:25.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.275+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:25.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.443+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:25.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.581+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:25.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.581+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:25.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.600+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:25.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.600+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:25.603 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.600+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:25.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.643+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:25.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.643+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:25.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.643+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:25.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.685+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:25.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.697+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:25.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.697+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:25.707 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:25.703+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:27.218 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:27.217+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:27.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:27.224+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:27.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:27.224+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:27.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:27.347+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:27.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:27.388+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:28.278 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.277+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:28.287 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.287+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:28.287 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.287+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:28.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.305+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:28.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.312+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:28.317 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.312+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:28.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:28.350+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:29.477 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:28:29.897 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:29.893+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:29.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:29.907+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:30.189 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:28:30.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.190+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:30.208 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:28:30.235 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2060,"stamp":"2023-10-03T18:28:29.402792+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23883986737,"num_objects":68542,"num_object_clones":63276,"num_object_copies":137084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68542,"num_whiteouts":0,"num_read":1459707,"num_read_kb":5282496,"num_write":1319683,"num_write_kb":5261115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80525,"ondisk_log_size":80525,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14238432,"kb_used_data":10523784,"kb_used_omap":9077,"kb_used_meta":3705546,"kb_avail":268877088,"statfs":{"total":289910292480,"available":275330138112,"internally_reserved":0,"allocated":10776354816,"data_stored":123238846195,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":3794479280},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,12,14,15,3,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10944512,"num_objects":526,"num_object_clones":526,"num_object_copies":1052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":526,"num_whiteouts":0,"num_read":2595,"num_read_kb":10344,"num_write":2641,"num_write_kb":10564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001689"},"pg_stats":[{"pgid":"2.7","version":"397'199514","reported_seq":350775,"reported_epoch":397,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:27.864296+0000","last_change":"2023-10-03T18:28:26.856066+0000","last_active":"2023-10-03T18:28:27.864296+0000","last_peered":"2023-10-03T18:28:27.864296+0000","last_clean":"2023-10-03T18:28:27.864296+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:28:27.864296+0000","last_undegraded":"2023-10-03T18:28:27.864296+0000","last_fullsized":"2023-10-03T18:28:27.864296+0000","mapping_epoch":12,"log_start":"349'189477","ondisk_log_start":"349'189477","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10037,"log_dups_size":0,"ondisk_log_size":10037,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":305,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:48:10.941013+0000","scrub_duration":505.38444223300002,"objects_trimmed":20,"snaptrim_duration":0.66886894699999999,"stat_sum":{"num_bytes":2975285733,"num_objects":12290,"num_object_clones":11639,"num_object_copies":24580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12290,"num_whiteouts":0,"num_read":177943,"num_read_kb":657330,"num_write":163966,"num_write_kb":653909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19d","length":"1"}]},{"pgid":"2.6","version":"397'184572","reported_seq":325229,"reported_epoch":397,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:27.895086+0000","last_change":"2023-10-03T18:28:24.049852+0000","last_active":"2023-10-03T18:28:27.895086+0000","last_peered":"2023-10-03T18:28:27.895086+0000","last_clean":"2023-10-03T18:28:27.895086+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:28:27.895086+0000","last_undegraded":"2023-10-03T18:28:27.895086+0000","last_fullsized":"2023-10-03T18:28:27.895086+0000","mapping_epoch":12,"log_start":"346'174545","ondisk_log_start":"346'174545","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":220,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":15,"snaptrim_duration":1.4631637930000001,"stat_sum":{"num_bytes":2748969532,"num_objects":8946,"num_object_clones":8343,"num_object_copies":17892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8946,"num_whiteouts":0,"num_read":165039,"num_read_kb":606394,"num_write":151481,"num_write_kb":604608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"}]},{"pgid":"2.5","version":"397'221632","reported_seq":387488,"reported_epoch":397,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:27.847442+0000","last_change":"2023-10-03T18:28:26.148577+0000","last_active":"2023-10-03T18:28:27.847442+0000","last_peered":"2023-10-03T18:28:27.847442+0000","last_clean":"2023-10-03T18:28:27.847442+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:28:27.847442+0000","last_undegraded":"2023-10-03T18:28:27.847442+0000","last_fullsized":"2023-10-03T18:28:27.847442+0000","mapping_epoch":12,"log_start":"351'211551","ondisk_log_start":"351'211551","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":252,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":8.5371857339999995,"stat_sum":{"num_bytes":3224949440,"num_objects":11534,"num_object_clones":10823,"num_object_copies":23068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11534,"num_whiteouts":0,"num_read":203667,"num_read_kb":718251,"num_write":178722,"num_write_kb":711353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"8"}]},{"pgid":"2.0","version":"397'216477","reported_seq":372738,"reported_epoch":397,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:24.840963+0000","last_change":"2023-10-03T18:28:24.409803+0000","last_active":"2023-10-03T18:28:24.840963+0000","last_peered":"2023-10-03T18:28:24.840963+0000","last_clean":"2023-10-03T18:28:24.840963+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:28:24.840963+0000","last_undegraded":"2023-10-03T18:28:24.840963+0000","last_fullsized":"2023-10-03T18:28:24.840963+0000","mapping_epoch":12,"log_start":"364'206418","ondisk_log_start":"364'206418","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":189,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:06:54.727843+0000","scrub_duration":734.24538164700004,"objects_trimmed":29,"snaptrim_duration":1.7740369469999999,"stat_sum":{"num_bytes":3114386064,"num_objects":9114,"num_object_clones":8429,"num_object_copies":18228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9114,"num_whiteouts":0,"num_read":184934,"num_read_kb":684381,"num_write":172139,"num_write_kb":686610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":562,"reported_epoch":396,"state":"active+clean","last_fresh":"2023-10-03T18:28:12.772336+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:28:12.772336+0000","last_peered":"2023-10-03T18:28:12.772336+0000","last_clean":"2023-10-03T18:28:12.772336+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:28:12.772336+0000","last_undegraded":"2023-10-03T18:28:12.772336+0000","last_fullsized":"2023-10-03T18:28:12.772336+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"397'213063","reported_seq":372300,"reported_epoch":397,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:27.809523+0000","last_change":"2023-10-03T18:28:26.093714+0000","last_active":"2023-10-03T18:28:27.809523+0000","last_peered":"2023-10-03T18:28:27.809523+0000","last_clean":"2023-10-03T18:28:27.809523+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:28:27.809523+0000","last_undegraded":"2023-10-03T18:28:27.809523+0000","last_fullsized":"2023-10-03T18:28:27.809523+0000","mapping_epoch":12,"log_start":"349'203059","ondisk_log_start":"349'203059","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10004,"log_dups_size":0,"ondisk_log_size":10004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":242,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":36,"snaptrim_duration":3.5109080820000003,"stat_sum":{"num_bytes":3167597386,"num_objects":11001,"num_object_clones":10302,"num_object_copies":22002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11001,"num_whiteouts":0,"num_read":188490,"num_read_kb":697074,"num_write":174557,"num_write_kb":696202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"8"}]},{"pgid":"2.1","version":"397'209720","reported_seq":354057,"reported_epoch":397,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:24.828287+0000","last_change":"2023-10-03T18:28:23.598605+0000","last_active":"2023-10-03T18:28:24.828287+0000","last_peered":"2023-10-03T18:28:24.828287+0000","last_clean":"2023-10-03T18:28:24.828287+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:28:24.828287+0000","last_undegraded":"2023-10-03T18:28:24.828287+0000","last_fullsized":"2023-10-03T18:28:24.828287+0000","mapping_epoch":12,"log_start":"352'199658","ondisk_log_start":"352'199658","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10062,"log_dups_size":0,"ondisk_log_size":10062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:46:35.935641+0000","scrub_duration":424.08306342600002,"objects_trimmed":36,"snaptrim_duration":0.99383976500000004,"stat_sum":{"num_bytes":2886410863,"num_objects":3033,"num_object_clones":2398,"num_object_copies":6066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3033,"num_whiteouts":0,"num_read":177786,"num_read_kb":636938,"num_write":159406,"num_write_kb":634975,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"1"}]},{"pgid":"2.2","version":"397'209863","reported_seq":359553,"reported_epoch":397,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:26.402687+0000","last_change":"2023-10-03T18:28:24.344516+0000","last_active":"2023-10-03T18:28:26.402687+0000","last_peered":"2023-10-03T18:28:26.402687+0000","last_clean":"2023-10-03T18:28:26.402687+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:28:26.402687+0000","last_undegraded":"2023-10-03T18:28:26.402687+0000","last_fullsized":"2023-10-03T18:28:26.402687+0000","mapping_epoch":12,"log_start":"352'199802","ondisk_log_start":"352'199802","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:55:10.868861+0000","scrub_duration":370.574046178,"objects_trimmed":50,"snaptrim_duration":1.7291269470000001,"stat_sum":{"num_bytes":2889573125,"num_objects":3085,"num_object_clones":2443,"num_object_copies":6170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":187807,"num_read_kb":648636,"num_write":160706,"num_write_kb":639275,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"}]},{"pgid":"2.4","version":"397'194937","reported_seq":341725,"reported_epoch":397,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:27.841059+0000","last_change":"2023-10-03T18:28:26.855295+0000","last_active":"2023-10-03T18:28:27.841059+0000","last_peered":"2023-10-03T18:28:27.841059+0000","last_clean":"2023-10-03T18:28:27.841059+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:28:27.841059+0000","last_undegraded":"2023-10-03T18:28:27.841059+0000","last_fullsized":"2023-10-03T18:28:27.841059+0000","mapping_epoch":12,"log_start":"348'184842","ondisk_log_start":"348'184842","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":226,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":4.2724185009999998,"stat_sum":{"num_bytes":2876224226,"num_objects":9537,"num_object_clones":8899,"num_object_copies":19074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9537,"num_whiteouts":0,"num_read":173935,"num_read_kb":633401,"num_write":158569,"num_write_kb":632289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"19d","length":"1"},{"start":"19f","length":"1"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23883396369,"num_objects":68540,"num_object_clones":63276,"num_object_copies":137080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68540,"num_whiteouts":0,"num_read":1459601,"num_read_kb":5282405,"num_write":1319546,"num_write_kb":5259221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10762649600,"data_stored":123254303266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80426,"ondisk_log_size":80426,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739036,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3417752,"kb_used_data":2020368,"kb_used_omap":1375,"kb_used_meta":1396000,"kb_avail":90954088,"statfs":{"total":96636764160,"available":93136986112,"internally_reserved":0,"allocated":2068856832,"data_stored":20756144932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1429504132},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5569999999999999,"5min":2.3319999999999999,"15min":2.5880000000000001},"min":{"1min":0.19700000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":11.4,"5min":12.737,"15min":18.321000000000002},"last":0.246},{"interface":"front","average":{"1min":2.2010000000000001,"5min":2.6930000000000001,"15min":2.4590000000000001},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":9.6259999999999994,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.90000000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:27:26 2023","interfaces":[{"interface":"back","average":{"1min":2.6160000000000001,"5min":2.6989999999999998,"15min":2.5390000000000001},"min":{"1min":0.255,"5min":0.25,"15min":0.221},"max":{"1min":9.5920000000000005,"5min":15.85,"15min":18.768999999999998},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.849,"5min":2.423,"15min":2.3479999999999999},"min":{"1min":0.23300000000000001,"5min":0.215,"15min":0.19700000000000001},"max":{"1min":8.3740000000000006,"5min":13.842000000000001,"15min":18.751999999999999},"last":0.55400000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739035,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4033404,"kb_used_data":3242868,"kb_used_omap":2774,"kb_used_meta":787753,"kb_avail":90338436,"statfs":{"total":96636764160,"available":92506558464,"internally_reserved":0,"allocated":3320696832,"data_stored":40858700227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":806659679},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,5,1,1,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":4.8979999999999997,"5min":3.758,"15min":3.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":21.256,"5min":45.886000000000003,"15min":45.886000000000003},"last":15.765000000000001},{"interface":"front","average":{"1min":3.8519999999999999,"5min":3.29,"15min":3.2770000000000001},"min":{"1min":0.20399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":19.018999999999998,"5min":45.555,"15min":45.555},"last":8.1809999999999992}]},{"osd":2,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.113,"15min":2.0049999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":8.9860000000000007,"5min":11.252000000000001,"15min":13.209},"last":0.66400000000000003},{"interface":"front","average":{"1min":5.4249999999999998,"5min":3,"15min":2.5230000000000001},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":25.77,"5min":25.77,"15min":25.77},"last":8.2170000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739007,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6787276,"kb_used_data":5260548,"kb_used_omap":4927,"kb_used_meta":1521792,"kb_avail":87584564,"statfs":{"total":96636764160,"available":89686593536,"internally_reserved":0,"allocated":5386801152,"data_stored":61624001036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1558315469},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,3,9,5,11,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.3170000000000002,"5min":3.3799999999999999,"15min":3.274},"min":{"1min":0.22,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":20.466999999999999,"5min":22.068999999999999,"15min":30.896999999999998},"last":2.2490000000000001},{"interface":"front","average":{"1min":5.0140000000000002,"5min":3.2589999999999999,"15min":3.044},"min":{"1min":0.26700000000000002,"5min":0.23100000000000001,"15min":0.17799999999999999},"max":{"1min":20.436,"5min":24.917000000000002,"15min":24.917000000000002},"last":0.372}]},{"osd":2,"last update":"Tue Oct 3 18:27:37 2023","interfaces":[{"interface":"back","average":{"1min":5.21,"5min":2.8860000000000001,"15min":2.3069999999999999},"min":{"1min":0.34000000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":34.395000000000003,"5min":34.395000000000003,"15min":34.395000000000003},"last":2.2109999999999999},{"interface":"front","average":{"1min":5.5259999999999998,"5min":3.153,"15min":2.9870000000000001},"min":{"1min":0.26700000000000002,"5min":0.186,"15min":0.186},"max":{"1min":36.332999999999998,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.85299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3315826688,"data_stored":40863247560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5382361088,"data_stored":61645784337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2064461824,"data_stored":20745271369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:28:30.236 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:30.237 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:30.237 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:30.237 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:30.237 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:28:30.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:30.348 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:30.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:30.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.345+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:30.380 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.380+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:30.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.391+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:30.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.419+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:30.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.419+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:30.473 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.472+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:30.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.591+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:30.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:30.599+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:34.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.066+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:34.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.094+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:34.095 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.094+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:34.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.136+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:34.153 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:34.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.212+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:34.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.811+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:34.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.936+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:34.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:34.936+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:35.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.091+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:35.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.091+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:35.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.091+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:35.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.091+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:35.092 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.091+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:35.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.442+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:35.443 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.442+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:35.489 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.488+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:35.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.506+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:35.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.568+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:35.571 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.568+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:35.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.591+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:35.592 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.591+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:35.622 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:35.620+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:36.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.327+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:36.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.350+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:36.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.350+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:36.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:36.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:36.358 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:36.351+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:37.821 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:37.820+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:37.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:37.858+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:37.861 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:37.860+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:38.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:38.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:38.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:38.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:38.186 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:38.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:38.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:38.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.184+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:38.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.479+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:38.480 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.479+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:38.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.479+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:38.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.479+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:38.504 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.498+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:38.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.518+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:38.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.518+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:38.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.518+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:38.522 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.518+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:38.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.550+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:38.550 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.550+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:38.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.881+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:38.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.885+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:38.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.885+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:38.925 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.921+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:38.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.925+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:38.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:38.925+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:39.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:39.013+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:39.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:39.043+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:39.045 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:39.043+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:40.350 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:40.350+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:40.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:40.350+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:40.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:40.412+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:42.660 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.660+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:42.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.660+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:42.694 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.694+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:42.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.721+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:42.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.874+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:42.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.874+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:42.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.874+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:42.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.874+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:42.878 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:42.876+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:43.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.319+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:43.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.359+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:43.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.359+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:43.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.423+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:43.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.423+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:43.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.423+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:43.432 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.423+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:43.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.449+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:43.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:43.451+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:44.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.554+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:44.585 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.584+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:44.599 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.598+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:44.599 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.598+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:44.620 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.619+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:44.621 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:44.619+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:49.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.566+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:28:49.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.569+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:49.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.569+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:49.570 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.569+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:49.579 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.572+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:49.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.623+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:49.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.679+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:49.680 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.679+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:49.911 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.910+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:49.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.918+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:49.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:49.925+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:50.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:28:50.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:28:50.980 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:28:51.012 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2074,"stamp":"2023-10-03T18:28:49.416146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23904786225,"num_objects":68443,"num_object_clones":63177,"num_object_copies":136886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68443,"num_whiteouts":0,"num_read":1464422,"num_read_kb":5301268,"num_write":1324739,"num_write_kb":5281339,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80536,"ondisk_log_size":80536,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14539108,"kb_used_data":10565380,"kb_used_omap":9077,"kb_used_meta":3964618,"kb_avail":268576412,"statfs":{"total":289910292480,"available":275022245888,"internally_reserved":0,"allocated":10818949120,"data_stored":123404412291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":4059769008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,2,8,7,15,12],"upper_bound":256},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7643136,"num_objects":-79,"num_object_clones":-79,"num_object_copies":-158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-79,"num_whiteouts":0,"num_read":1744,"num_read_kb":6948,"num_write":1858,"num_write_kb":7432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.006462"},"pg_stats":[{"pgid":"2.7","version":"400'200375","reported_seq":352035,"reported_epoch":400,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:44.296916+0000","last_change":"2023-10-03T18:28:43.084710+0000","last_active":"2023-10-03T18:28:44.296916+0000","last_peered":"2023-10-03T18:28:44.296916+0000","last_clean":"2023-10-03T18:28:44.296916+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:28:44.296916+0000","last_undegraded":"2023-10-03T18:28:44.296916+0000","last_fullsized":"2023-10-03T18:28:44.296916+0000","mapping_epoch":12,"log_start":"351'190277","ondisk_log_start":"351'190277","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":302,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:48:10.941013+0000","scrub_duration":505.38444223300002,"objects_trimmed":18,"snaptrim_duration":2.9339983620000001,"stat_sum":{"num_bytes":2977563109,"num_objects":12316,"num_object_clones":11665,"num_object_copies":24632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12316,"num_whiteouts":0,"num_read":178465,"num_read_kb":659418,"num_write":164515,"num_write_kb":656105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a4","length":"1"}]},{"pgid":"2.6","version":"400'185416","reported_seq":326464,"reported_epoch":400,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:44.223480+0000","last_change":"2023-10-03T18:28:42.427795+0000","last_active":"2023-10-03T18:28:44.223480+0000","last_peered":"2023-10-03T18:28:44.223480+0000","last_clean":"2023-10-03T18:28:44.223480+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:28:44.223480+0000","last_undegraded":"2023-10-03T18:28:44.223480+0000","last_fullsized":"2023-10-03T18:28:44.223480+0000","mapping_epoch":12,"log_start":"348'175350","ondisk_log_start":"348'175350","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":217,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.8300596120000003,"stat_sum":{"num_bytes":2751201852,"num_objects":9000,"num_object_clones":8397,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9000,"num_whiteouts":0,"num_read":165574,"num_read_kb":608530,"num_write":152026,"num_write_kb":606788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"3"}]},{"pgid":"2.5","version":"400'222575","reported_seq":388902,"reported_epoch":400,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:44.224810+0000","last_change":"2023-10-03T18:28:43.064924+0000","last_active":"2023-10-03T18:28:44.224810+0000","last_peered":"2023-10-03T18:28:44.224810+0000","last_clean":"2023-10-03T18:28:44.224810+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:28:44.224810+0000","last_undegraded":"2023-10-03T18:28:44.224810+0000","last_fullsized":"2023-10-03T18:28:44.224810+0000","mapping_epoch":12,"log_start":"355'212555","ondisk_log_start":"355'212555","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10020,"log_dups_size":0,"ondisk_log_size":10020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":249,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.454449993,"stat_sum":{"num_bytes":3227476672,"num_objects":11586,"num_object_clones":10875,"num_object_copies":23172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11586,"num_whiteouts":0,"num_read":204265,"num_read_kb":720635,"num_write":179339,"num_write_kb":713821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"1"}]},{"pgid":"2.0","version":"400'218035","reported_seq":374678,"reported_epoch":400,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:45.047919+0000","last_change":"2023-10-03T18:28:44.907270+0000","last_active":"2023-10-03T18:28:45.047919+0000","last_peered":"2023-10-03T18:28:45.047919+0000","last_clean":"2023-10-03T18:28:45.047919+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:28:45.047919+0000","last_undegraded":"2023-10-03T18:28:45.047919+0000","last_fullsized":"2023-10-03T18:28:45.047919+0000","mapping_epoch":12,"log_start":"367'208026","ondisk_log_start":"367'208026","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10009,"log_dups_size":0,"ondisk_log_size":10009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":180,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:06:54.727843+0000","scrub_duration":734.24538164700004,"objects_trimmed":42,"snaptrim_duration":2.3065165169999999,"stat_sum":{"num_bytes":3117568656,"num_objects":8980,"num_object_clones":8295,"num_object_copies":17960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8980,"num_whiteouts":0,"num_read":185596,"num_read_kb":687025,"num_write":172910,"num_write_kb":689694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":565,"reported_epoch":399,"state":"active+clean","last_fresh":"2023-10-03T18:28:37.626156+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:28:37.626156+0000","last_peered":"2023-10-03T18:28:37.626156+0000","last_clean":"2023-10-03T18:28:37.626156+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:28:37.626156+0000","last_undegraded":"2023-10-03T18:28:37.626156+0000","last_fullsized":"2023-10-03T18:28:37.626156+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"400'214037","reported_seq":373652,"reported_epoch":400,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:28:44.164783+0000","last_change":"2023-10-03T18:28:40.780931+0000","last_active":"2023-10-03T18:28:44.164783+0000","last_peered":"2023-10-03T18:28:44.164783+0000","last_clean":"2023-10-03T18:28:44.164783+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:28:44.164783+0000","last_undegraded":"2023-10-03T18:28:44.164783+0000","last_fullsized":"2023-10-03T18:28:44.164783+0000","mapping_epoch":12,"log_start":"352'203959","ondisk_log_start":"352'203959","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":240,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.172602243,"stat_sum":{"num_bytes":3170329418,"num_objects":11090,"num_object_clones":10391,"num_object_copies":22180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11090,"num_whiteouts":0,"num_read":189038,"num_read_kb":699254,"num_write":175216,"num_write_kb":698838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"}]},{"pgid":"2.1","version":"400'211050","reported_seq":355841,"reported_epoch":400,"state":"active+clean","last_fresh":"2023-10-03T18:28:45.054085+0000","last_change":"2023-10-03T18:28:40.463979+0000","last_active":"2023-10-03T18:28:45.054085+0000","last_peered":"2023-10-03T18:28:45.054085+0000","last_clean":"2023-10-03T18:28:45.054085+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:28:45.054085+0000","last_undegraded":"2023-10-03T18:28:45.054085+0000","last_fullsized":"2023-10-03T18:28:45.054085+0000","mapping_epoch":12,"log_start":"364'200963","ondisk_log_start":"364'200963","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:46:35.935641+0000","scrub_duration":424.08306342600002,"objects_trimmed":46,"snaptrim_duration":2.8515251020000001,"stat_sum":{"num_bytes":2889171567,"num_objects":2942,"num_object_clones":2307,"num_object_copies":5884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":178459,"num_read_kb":639602,"num_write":160081,"num_write_kb":637675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"}]},{"pgid":"2.2","version":"400'211260","reported_seq":361379,"reported_epoch":400,"state":"active+clean","last_fresh":"2023-10-03T18:28:46.594544+0000","last_change":"2023-10-03T18:28:43.926037+0000","last_active":"2023-10-03T18:28:46.594544+0000","last_peered":"2023-10-03T18:28:46.594544+0000","last_clean":"2023-10-03T18:28:46.594544+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:28:46.594544+0000","last_undegraded":"2023-10-03T18:28:46.594544+0000","last_fullsized":"2023-10-03T18:28:46.594544+0000","mapping_epoch":12,"log_start":"364'201208","ondisk_log_start":"364'201208","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10052,"log_dups_size":0,"ondisk_log_size":10052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:55:10.868861+0000","scrub_duration":370.574046178,"objects_trimmed":22,"snaptrim_duration":1.277577733,"stat_sum":{"num_bytes":2892485381,"num_objects":2962,"num_object_clones":2320,"num_object_copies":5924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":188448,"num_read_kb":651184,"num_write":161415,"num_write_kb":642111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"}]},{"pgid":"2.4","version":"400'195773","reported_seq":342965,"reported_epoch":400,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:28:44.185370+0000","last_change":"2023-10-03T18:28:42.463168+0000","last_active":"2023-10-03T18:28:44.185370+0000","last_peered":"2023-10-03T18:28:44.185370+0000","last_clean":"2023-10-03T18:28:44.185370+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:28:44.185370+0000","last_undegraded":"2023-10-03T18:28:44.185370+0000","last_fullsized":"2023-10-03T18:28:44.185370+0000","mapping_epoch":12,"log_start":"350'185746","ondisk_log_start":"350'185746","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":224,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":1.0449085739999999,"stat_sum":{"num_bytes":2878399202,"num_objects":9565,"num_object_clones":8927,"num_object_copies":19130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9565,"num_whiteouts":0,"num_read":174471,"num_read_kb":635529,"num_write":159100,"num_write_kb":634413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1a4","length":"1"},{"start":"1a6","length":"1"},{"start":"1a8","length":"1"},{"start":"1ae","length":"a"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23904195857,"num_objects":68441,"num_object_clones":63177,"num_object_copies":136882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68441,"num_whiteouts":0,"num_read":1464316,"num_read_kb":5301177,"num_write":1324602,"num_write_kb":5279445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10805444608,"data_stored":123396545058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80437,"ondisk_log_size":80437,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739040,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3477092,"kb_used_data":2029144,"kb_used_omap":1375,"kb_used_meta":1446560,"kb_avail":90894748,"statfs":{"total":96636764160,"available":93076221952,"internally_reserved":0,"allocated":2077843456,"data_stored":20701365588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1481277572},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,1,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:28:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":2.508,"15min":2.585},"min":{"1min":0.188,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":14.26,"5min":14.26,"15min":18.321000000000002},"last":3.286},{"interface":"front","average":{"1min":2.1920000000000002,"5min":2.7839999999999998,"15min":2.4489999999999998},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":12.082000000000001,"5min":34.176000000000002,"15min":34.176000000000002},"last":3.2080000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:28:26 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.7869999999999999,"15min":2.4940000000000002},"min":{"1min":0.34999999999999998,"5min":0.25,"15min":0.221},"max":{"1min":13.836,"5min":15.85,"15min":17.274000000000001},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.516,"5min":2.379,"15min":2.2519999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.7080000000000002,"5min":13.842000000000001,"15min":13.842000000000001},"last":1.4610000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739039,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4168124,"kb_used_data":3256820,"kb_used_omap":2774,"kb_used_meta":908521,"kb_avail":90203716,"statfs":{"total":96636764160,"available":92368605184,"internally_reserved":0,"allocated":3334983680,"data_stored":41007643635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":930326111},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":4.8979999999999997,"5min":3.758,"15min":3.1619999999999999},"min":{"1min":0.30099999999999999,"5min":0.184,"15min":0.14799999999999999},"max":{"1min":21.256,"5min":45.886000000000003,"15min":45.886000000000003},"last":0.65500000000000003},{"interface":"front","average":{"1min":3.8519999999999999,"5min":3.29,"15min":3.2770000000000001},"min":{"1min":0.20399999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":19.018999999999998,"5min":45.555,"15min":45.555},"last":8.0549999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:27:44 2023","interfaces":[{"interface":"back","average":{"1min":2.4660000000000002,"5min":2.113,"15min":2.0049999999999999},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":8.9860000000000007,"5min":11.252000000000001,"15min":13.209},"last":0.63400000000000001},{"interface":"front","average":{"1min":5.4249999999999998,"5min":3,"15min":2.5230000000000001},"min":{"1min":0.215,"5min":0.192,"15min":0.192},"max":{"1min":25.77,"5min":25.77,"15min":25.77},"last":0.71599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739010,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6893892,"kb_used_data":5279416,"kb_used_omap":4927,"kb_used_meta":1609536,"kb_avail":87477948,"statfs":{"total":96636764160,"available":89577418752,"internally_reserved":0,"allocated":5406121984,"data_stored":61695403068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1648165325},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,5,4,14,6],"upper_bound":256},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":3.2519999999999998,"15min":3.2989999999999999},"min":{"1min":0.32100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.8369999999999997,"5min":20.466999999999999,"15min":30.896999999999998},"last":5.0640000000000001},{"interface":"front","average":{"1min":1.3640000000000001,"5min":2.9729999999999999,"15min":2.8969999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17799999999999999},"max":{"1min":4.5510000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":8.0109999999999992}]},{"osd":2,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.8959999999999999,"15min":2.2789999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.9279999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.29799999999999999},{"interface":"front","average":{"1min":1.827,"5min":2.9180000000000001,"15min":2.8479999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":4.9429999999999996,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.27800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3330322432,"data_stored":41009335496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5401468928,"data_stored":61690787089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2073653248,"data_stored":20696422473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:28:51.014 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:51.014 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:51.015 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:51.015 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:28:51.015 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:28:51.751 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:28:51.771 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:28:52.483 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:28:52.498 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:28:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:28:53.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:28:53.981 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:28:53.997 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:28:55.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.788+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:55.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.788+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:55.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.802+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:55.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.802+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:55.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.802+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:55.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.874+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:55.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:55.917+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:56.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.345+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:56.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.377+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:56.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.393+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:56.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.393+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:56.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.393+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:56.416 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.413+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:56.424 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:56.424+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:57.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.655+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:28:57.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.656+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:57.660 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.659+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:57.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.690+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:57.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.705+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:57.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.711+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:57.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.776+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:57.784 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:57.782+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:28:59.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.656+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:28:59.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.656+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:28:59.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.656+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:28:59.715 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.704+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:28:59.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.726+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:28:59.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.726+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:28:59.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.779+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:28:59.780 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.779+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:28:59.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:28:59.957+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:02.099 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.098+0000 7fc515f98700 -1 reset not still connected to 0x5622229b6dd0 2023-10-03T18:29:02.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.123+0000 7fc515f98700 -1 reset not still connected to 0x56222d6552b0 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.138+0000 7fc515f98700 -1 reset not still connected to 0x562233dc5040 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.207+0000 7fc515f98700 -1 reset not still connected to 0x562234814dd0 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.616+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.616+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.616+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.734+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:02.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.734+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:02.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.735+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:02.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.735+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:02.855 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:02.780+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:03.692 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.685+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:29:03.693 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.688+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:03.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:03.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:03.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.804+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:03.844 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.808+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:03.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.884+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:03.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.884+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:03.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.884+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:03.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.885+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:03.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:03.957+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:04.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:04.817+0000 7fc515f98700 -1 reset not still connected to 0x5622168f8410 2023-10-03T18:29:04.824 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:04.823+0000 7fc515f98700 -1 reset not still connected to 0x5622280f0f70 2023-10-03T18:29:04.856 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:04.855+0000 7f9f21f20700 -1 reset not still connected to 0x5632f429c680 2023-10-03T18:29:09.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:09.116+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:09.147 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:09.142+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:09.955 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:09.954+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:29:09.956 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:09.954+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:29:09.956 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:09.954+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:29:09.956 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:09.954+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:29:09.956 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:09.954+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:29:13.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:29:14.722 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:29:14.761 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:29:14.776 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2096,"stamp":"2023-10-03T18:29:13.430646+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23936923441,"num_objects":69068,"num_object_clones":63802,"num_object_copies":138136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69068,"num_whiteouts":0,"num_read":1476521,"num_read_kb":5335287,"num_write":1332789,"num_write_kb":5312819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80625,"ondisk_log_size":80625,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14950480,"kb_used_data":10627376,"kb_used_omap":9077,"kb_used_meta":4313994,"kb_avail":268165040,"statfs":{"total":289910292480,"available":274601000960,"internally_reserved":0,"allocated":10882433024,"data_stored":125010461821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":4417530032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,10,12,14,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4976590,"num_objects":603,"num_object_clones":604,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":0,"num_read":1627,"num_read_kb":5172,"num_write":1258,"num_write_kb":4894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.762666"},"pg_stats":[{"pgid":"2.7","version":"410'201919","reported_seq":354417,"reported_epoch":410,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:10.946456+0000","last_change":"2023-10-03T18:29:08.333522+0000","last_active":"2023-10-03T18:29:10.946456+0000","last_peered":"2023-10-03T18:29:10.946456+0000","last_clean":"2023-10-03T18:29:10.946456+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:29:10.946456+0000","last_undegraded":"2023-10-03T18:29:10.946456+0000","last_fullsized":"2023-10-03T18:29:10.946456+0000","mapping_epoch":12,"log_start":"364'191883","ondisk_log_start":"364'191883","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":317,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:29:13.880240+0000","scrub_duration":505.38444223300002,"objects_trimmed":21,"snaptrim_duration":1.6016838410000001,"stat_sum":{"num_bytes":2981659059,"num_objects":12425,"num_object_clones":11775,"num_object_copies":24850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12425,"num_whiteouts":0,"num_read":179888,"num_read_kb":663766,"num_write":165535,"num_write_kb":660116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1af","length":"2"}]},{"pgid":"2.6","version":"410'186870","reported_seq":328721,"reported_epoch":410,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:10.970390+0000","last_change":"2023-10-03T18:29:09.205945+0000","last_active":"2023-10-03T18:29:10.970390+0000","last_peered":"2023-10-03T18:29:10.970390+0000","last_clean":"2023-10-03T18:29:10.970390+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:29:10.970390+0000","last_undegraded":"2023-10-03T18:29:10.970390+0000","last_fullsized":"2023-10-03T18:29:10.970390+0000","mapping_epoch":12,"log_start":"353'176857","ondisk_log_start":"353'176857","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":233,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0.88074495600000002,"stat_sum":{"num_bytes":2755097199,"num_objects":9091,"num_object_clones":8487,"num_object_copies":18182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9091,"num_whiteouts":0,"num_read":166832,"num_read_kb":612387,"num_write":152988,"num_write_kb":610597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"7"}]},{"pgid":"2.5","version":"410'224239","reported_seq":391492,"reported_epoch":410,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:10.821948+0000","last_change":"2023-10-03T18:29:08.347612+0000","last_active":"2023-10-03T18:29:10.821948+0000","last_peered":"2023-10-03T18:29:10.821948+0000","last_clean":"2023-10-03T18:29:10.821948+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:29:10.821948+0000","last_undegraded":"2023-10-03T18:29:10.821948+0000","last_fullsized":"2023-10-03T18:29:10.821948+0000","mapping_epoch":12,"log_start":"365'214159","ondisk_log_start":"365'214159","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10080,"log_dups_size":0,"ondisk_log_size":10080,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":266,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":18,"snaptrim_duration":0.17593299600000001,"stat_sum":{"num_bytes":3231924877,"num_objects":11726,"num_object_clones":11016,"num_object_copies":23452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11726,"num_whiteouts":0,"num_read":206118,"num_read_kb":725344,"num_write":180463,"num_write_kb":718169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"4"}]},{"pgid":"2.0","version":"410'220009","reported_seq":377379,"reported_epoch":410,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:10.154543+0000","last_change":"2023-10-03T18:29:10.123700+0000","last_active":"2023-10-03T18:29:10.154543+0000","last_peered":"2023-10-03T18:29:10.154543+0000","last_clean":"2023-10-03T18:29:10.154543+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:29:10.154543+0000","last_undegraded":"2023-10-03T18:29:10.154543+0000","last_fullsized":"2023-10-03T18:29:10.154543+0000","mapping_epoch":12,"log_start":"371'209939","ondisk_log_start":"371'209939","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":189,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:06:36.476597+0000","scrub_duration":734.24538164700004,"objects_trimmed":38,"snaptrim_duration":3.4367150180000001,"stat_sum":{"num_bytes":3121746726,"num_objects":8856,"num_object_clones":8168,"num_object_copies":17712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8856,"num_whiteouts":0,"num_read":186970,"num_read_kb":691160,"num_write":173949,"num_write_kb":693776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":575,"reported_epoch":409,"state":"active+clean","last_fresh":"2023-10-03T18:29:05.749990+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:29:05.749990+0000","last_peered":"2023-10-03T18:29:05.749990+0000","last_clean":"2023-10-03T18:29:05.749990+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:29:05.749990+0000","last_undegraded":"2023-10-03T18:29:05.749990+0000","last_fullsized":"2023-10-03T18:29:05.749990+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"410'215661","reported_seq":376072,"reported_epoch":410,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:10.674311+0000","last_change":"2023-10-03T18:29:09.146110+0000","last_active":"2023-10-03T18:29:10.674311+0000","last_peered":"2023-10-03T18:29:10.674311+0000","last_clean":"2023-10-03T18:29:10.674311+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:29:10.674311+0000","last_undegraded":"2023-10-03T18:29:10.674311+0000","last_fullsized":"2023-10-03T18:29:10.674311+0000","mapping_epoch":12,"log_start":"364'205562","ondisk_log_start":"364'205562","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":256,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":2.4377288340000001,"stat_sum":{"num_bytes":3174654845,"num_objects":11226,"num_object_clones":10526,"num_object_copies":22452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11226,"num_whiteouts":0,"num_read":190444,"num_read_kb":703511,"num_write":176296,"num_write_kb":703076,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"4"}]},{"pgid":"2.1","version":"410'212459","reported_seq":358063,"reported_epoch":410,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:10.137518+0000","last_change":"2023-10-03T18:29:09.810880+0000","last_active":"2023-10-03T18:29:10.137518+0000","last_peered":"2023-10-03T18:29:10.137518+0000","last_clean":"2023-10-03T18:29:10.137518+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:29:10.137518+0000","last_undegraded":"2023-10-03T18:29:10.137518+0000","last_fullsized":"2023-10-03T18:29:10.137518+0000","mapping_epoch":12,"log_start":"366'202370","ondisk_log_start":"366'202370","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:11:34.357820+0000","scrub_duration":424.08306342600002,"objects_trimmed":64,"snaptrim_duration":3.1267747039999998,"stat_sum":{"num_bytes":2892882441,"num_objects":3015,"num_object_clones":2382,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3015,"num_whiteouts":0,"num_read":179975,"num_read_kb":643636,"num_write":161017,"num_write_kb":641317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"4"}]},{"pgid":"2.2","version":"410'212710","reported_seq":363740,"reported_epoch":410,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:11.878013+0000","last_change":"2023-10-03T18:29:10.444812+0000","last_active":"2023-10-03T18:29:11.878013+0000","last_peered":"2023-10-03T18:29:11.878013+0000","last_clean":"2023-10-03T18:29:11.878013+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:29:11.878013+0000","last_undegraded":"2023-10-03T18:29:11.878013+0000","last_fullsized":"2023-10-03T18:29:11.878013+0000","mapping_epoch":12,"log_start":"367'202611","ondisk_log_start":"367'202611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:37:30.772170+0000","scrub_duration":370.574046178,"objects_trimmed":73,"snaptrim_duration":3.7391064599999999,"stat_sum":{"num_bytes":2896184120,"num_objects":3061,"num_object_clones":2418,"num_object_copies":6122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3061,"num_whiteouts":0,"num_read":190272,"num_read_kb":655637,"num_write":162358,"num_write_kb":645750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"4"}]},{"pgid":"2.4","version":"410'197197","reported_seq":345279,"reported_epoch":410,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:10.754868+0000","last_change":"2023-10-03T18:29:04.221432+0000","last_active":"2023-10-03T18:29:10.754868+0000","last_peered":"2023-10-03T18:29:10.754868+0000","last_clean":"2023-10-03T18:29:10.754868+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:29:10.754868+0000","last_undegraded":"2023-10-03T18:29:10.754868+0000","last_fullsized":"2023-10-03T18:29:10.754868+0000","mapping_epoch":12,"log_start":"358'187157","ondisk_log_start":"358'187157","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":239,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.532970626,"stat_sum":{"num_bytes":2882183806,"num_objects":9666,"num_object_clones":9030,"num_object_copies":19332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9666,"num_whiteouts":0,"num_read":175916,"num_read_kb":639755,"num_write":160046,"num_write_kb":638124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1af","length":"9"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23936333073,"num_objects":69066,"num_object_clones":63802,"num_object_copies":138132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69066,"num_whiteouts":0,"num_read":1476415,"num_read_kb":5335196,"num_write":1332652,"num_write_kb":5310925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10868936704,"data_stored":125182732834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80526,"ondisk_log_size":80526,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739045,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3551376,"kb_used_data":2040964,"kb_used_omap":1375,"kb_used_meta":1509024,"kb_avail":90820464,"statfs":{"total":96636764160,"available":93000155136,"internally_reserved":0,"allocated":2089947136,"data_stored":20825070261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1545240708},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:28:26 2023","interfaces":[{"interface":"back","average":{"1min":2.5350000000000001,"5min":2.508,"15min":2.585},"min":{"1min":0.188,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":14.26,"5min":14.26,"15min":18.321000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":2.1920000000000002,"5min":2.7839999999999998,"15min":2.4489999999999998},"min":{"1min":0.27600000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":12.082000000000001,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.30599999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:28:26 2023","interfaces":[{"interface":"back","average":{"1min":2.1520000000000001,"5min":2.7869999999999999,"15min":2.4940000000000002},"min":{"1min":0.34999999999999998,"5min":0.25,"15min":0.221},"max":{"1min":13.836,"5min":15.85,"15min":17.274000000000001},"last":0.371},{"interface":"front","average":{"1min":1.516,"5min":2.379,"15min":2.2519999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.7080000000000002,"5min":13.842000000000001,"15min":13.842000000000001},"last":0.39000000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739044,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4272720,"kb_used_data":3275336,"kb_used_omap":2774,"kb_used_meta":994601,"kb_avail":90099120,"statfs":{"total":96636764160,"available":92261498880,"internally_reserved":0,"allocated":3353944064,"data_stored":41729663118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1018472031},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:28:47 2023","interfaces":[{"interface":"back","average":{"1min":4.3639999999999999,"5min":4.1399999999999997,"15min":3.202},"min":{"1min":0.22700000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":19.873000000000001,"5min":45.886000000000003,"15min":45.886000000000003},"last":0.89300000000000002},{"interface":"front","average":{"1min":4.0439999999999996,"5min":3.6139999999999999,"15min":3.3279999999999998},"min":{"1min":0.26600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":12.581,"5min":45.555,"15min":45.555},"last":0.44}]},{"osd":2,"last update":"Tue Oct 3 18:28:47 2023","interfaces":[{"interface":"back","average":{"1min":1.944,"5min":2.1099999999999999,"15min":2},"min":{"1min":0.32500000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":7.7880000000000003,"5min":11.252000000000001,"15min":13.209},"last":6.8419999999999996},{"interface":"front","average":{"1min":3.254,"5min":3.1960000000000002,"15min":2.6040000000000001},"min":{"1min":0.26500000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.563000000000001,"5min":25.77,"15min":25.77},"last":0.40200000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739015,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7126384,"kb_used_data":5311076,"kb_used_omap":4927,"kb_used_meta":1810368,"kb_avail":87245456,"statfs":{"total":96636764160,"available":89339346944,"internally_reserved":0,"allocated":5438541824,"data_stored":62455728442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":1853817293},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,8,10,13,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":3.2519999999999998,"15min":3.2989999999999999},"min":{"1min":0.32100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.8369999999999997,"5min":20.466999999999999,"15min":30.896999999999998},"last":0.35499999999999998},{"interface":"front","average":{"1min":1.3640000000000001,"5min":2.9729999999999999,"15min":2.8969999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17799999999999999},"max":{"1min":4.5510000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":0.55100000000000005}]},{"osd":2,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.8959999999999999,"15min":2.2789999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.9279999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.432},{"interface":"front","average":{"1min":1.827,"5min":2.9180000000000001,"15min":2.8479999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":4.9429999999999996,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.59299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3348992000,"data_stored":41753345125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5434359808,"data_stored":62576153873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2085584896,"data_stored":20853233836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:29:14.778 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:14.778 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:14.778 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:14.778 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:14.778 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:29:19.713 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:19.711+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:29:19.713 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:19.711+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:29:19.713 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:19.711+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:29:23.187 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:23.185+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:29:23.245 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:23.243+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:29:23.245 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:23.243+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:29:23.245 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:29:23.243+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:29:23.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.357+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:23.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.357+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:23.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.480+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:23.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.490+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:23.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.490+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:23.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.529+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:23.559 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.557+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:23.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.695+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:23.710 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.708+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:29:23.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.708+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:23.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.708+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:23.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.708+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:23.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:23.708+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:31.147 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.140+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:31.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.187+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:31.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.202+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:31.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.274+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:31.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.303+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:31.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.303+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:31.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.367+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:31.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.392+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:31.414 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.412+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:31.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.463+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:29:31.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.499+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:31.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.506+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:31.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.506+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:31.508 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.506+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:31.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.521+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:31.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.521+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:31.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.521+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:31.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.592+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:31.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.599+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:31.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:31.599+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:32.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.110+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:32.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.126+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:32.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.126+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:32.128 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.126+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:32.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.185+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:32.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.203+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:32.211 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.203+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:32.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.309+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:32.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.309+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:32.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:32.309+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:33.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.086+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:33.239 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.237+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:33.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.245+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:33.251 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.251+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:33.320 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.320+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:33.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.327+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:33.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.327+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:33.329 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.328+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:33.329 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.328+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:33.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.328+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:33.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:33.342 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:33.343 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:33.344 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:33.342+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:34.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:29:35.535 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:29:35.541 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:29:35.560 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2110,"stamp":"2023-10-03T18:29:35.437094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23959160625,"num_objects":70111,"num_object_clones":64845,"num_object_copies":140222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70111,"num_whiteouts":0,"num_read":1481817,"num_read_kb":5356391,"num_write":1338209,"num_write_kb":5334499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80486,"ondisk_log_size":80486,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15275812,"kb_used_data":10670280,"kb_used_omap":9077,"kb_used_meta":4596426,"kb_avail":267839708,"statfs":{"total":289910292480,"available":274267860992,"internally_reserved":0,"allocated":10926366720,"data_stored":127720642525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":4706740400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,4,0,4,11,10,22,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10440704,"num_objects":228,"num_object_clones":228,"num_object_copies":456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":2481,"num_read_kb":9888,"num_write":2541,"num_write_kb":10164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003740"},"pg_stats":[{"pgid":"2.7","version":"414'203131","reported_seq":355882,"reported_epoch":414,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:32.947980+0000","last_change":"2023-10-03T18:29:30.963897+0000","last_active":"2023-10-03T18:29:32.947980+0000","last_peered":"2023-10-03T18:29:32.947980+0000","last_clean":"2023-10-03T18:29:32.947980+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:29:32.947980+0000","last_undegraded":"2023-10-03T18:29:32.947980+0000","last_fullsized":"2023-10-03T18:29:32.947980+0000","mapping_epoch":12,"log_start":"367'193086","ondisk_log_start":"367'193086","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10045,"log_dups_size":0,"ondisk_log_size":10045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":314,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:29:13.880240+0000","scrub_duration":505.38444223300002,"objects_trimmed":4,"snaptrim_duration":4.077798241,"stat_sum":{"num_bytes":2984350131,"num_objects":12601,"num_object_clones":11951,"num_object_copies":25202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12601,"num_whiteouts":0,"num_read":180472,"num_read_kb":666094,"num_write":166192,"num_write_kb":662744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"414'187910","reported_seq":330071,"reported_epoch":414,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:32.854865+0000","last_change":"2023-10-03T18:29:28.919192+0000","last_active":"2023-10-03T18:29:32.854865+0000","last_peered":"2023-10-03T18:29:32.854865+0000","last_clean":"2023-10-03T18:29:32.854865+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:29:32.854865+0000","last_undegraded":"2023-10-03T18:29:32.854865+0000","last_fullsized":"2023-10-03T18:29:32.854865+0000","mapping_epoch":12,"log_start":"364'177859","ondisk_log_start":"364'177859","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10051,"log_dups_size":0,"ondisk_log_size":10051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":229,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.275596808,"stat_sum":{"num_bytes":2757448303,"num_objects":9252,"num_object_clones":8648,"num_object_copies":18504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9252,"num_whiteouts":0,"num_read":167418,"num_read_kb":614719,"num_write":153563,"num_write_kb":612897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"1"}]},{"pgid":"2.5","version":"414'225454","reported_seq":393078,"reported_epoch":414,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:32.841986+0000","last_change":"2023-10-03T18:29:31.510307+0000","last_active":"2023-10-03T18:29:32.841986+0000","last_peered":"2023-10-03T18:29:32.841986+0000","last_clean":"2023-10-03T18:29:32.841986+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:29:32.841986+0000","last_undegraded":"2023-10-03T18:29:32.841986+0000","last_fullsized":"2023-10-03T18:29:32.841986+0000","mapping_epoch":12,"log_start":"369'215365","ondisk_log_start":"369'215365","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":262,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.871345313,"stat_sum":{"num_bytes":3234734733,"num_objects":11944,"num_object_clones":11234,"num_object_copies":23888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11944,"num_whiteouts":0,"num_read":206809,"num_read_kb":728092,"num_write":181149,"num_write_kb":720913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"8"}]},{"pgid":"2.0","version":"414'221859","reported_seq":379550,"reported_epoch":414,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:35.314255+0000","last_change":"2023-10-03T18:29:34.186717+0000","last_active":"2023-10-03T18:29:35.314255+0000","last_peered":"2023-10-03T18:29:35.314255+0000","last_clean":"2023-10-03T18:29:35.314255+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:29:35.314255+0000","last_undegraded":"2023-10-03T18:29:35.314255+0000","last_fullsized":"2023-10-03T18:29:35.314255+0000","mapping_epoch":12,"log_start":"381'211853","ondisk_log_start":"381'211853","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":179,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:06:36.476597+0000","scrub_duration":734.24538164700004,"objects_trimmed":35,"snaptrim_duration":1.5185476919999998,"stat_sum":{"num_bytes":3125068582,"num_objects":8891,"num_object_clones":8203,"num_object_copies":17782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8891,"num_whiteouts":0,"num_read":187765,"num_read_kb":694324,"num_write":174760,"num_write_kb":697020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":579,"reported_epoch":413,"state":"active+clean","last_fresh":"2023-10-03T18:29:27.661437+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:29:27.661437+0000","last_peered":"2023-10-03T18:29:27.661437+0000","last_clean":"2023-10-03T18:29:27.661437+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:29:27.661437+0000","last_undegraded":"2023-10-03T18:29:27.661437+0000","last_fullsized":"2023-10-03T18:29:27.661437+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"414'216898","reported_seq":377655,"reported_epoch":414,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:32.721802+0000","last_change":"2023-10-03T18:29:30.948898+0000","last_active":"2023-10-03T18:29:32.721802+0000","last_peered":"2023-10-03T18:29:32.721802+0000","last_clean":"2023-10-03T18:29:32.721802+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:29:32.721802+0000","last_undegraded":"2023-10-03T18:29:32.721802+0000","last_fullsized":"2023-10-03T18:29:32.721802+0000","mapping_epoch":12,"log_start":"367'206870","ondisk_log_start":"367'206870","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10028,"log_dups_size":0,"ondisk_log_size":10028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":253,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.3103017550000002,"stat_sum":{"num_bytes":3177718653,"num_objects":11478,"num_object_clones":10778,"num_object_copies":22956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11478,"num_whiteouts":0,"num_read":191129,"num_read_kb":706243,"num_write":177039,"num_write_kb":706048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"7"}]},{"pgid":"2.1","version":"414'214251","reported_seq":360126,"reported_epoch":414,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:35.355522+0000","last_change":"2023-10-03T18:29:34.285744+0000","last_active":"2023-10-03T18:29:35.355522+0000","last_peered":"2023-10-03T18:29:35.355522+0000","last_clean":"2023-10-03T18:29:35.355522+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:29:35.355522+0000","last_undegraded":"2023-10-03T18:29:35.355522+0000","last_fullsized":"2023-10-03T18:29:35.355522+0000","mapping_epoch":12,"log_start":"374'204184","ondisk_log_start":"374'204184","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:11:34.357820+0000","scrub_duration":424.08306342600002,"objects_trimmed":49,"snaptrim_duration":1.6397735419999999,"stat_sum":{"num_bytes":2895843849,"num_objects":2998,"num_object_clones":2365,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":180692,"num_read_kb":646496,"num_write":161737,"num_write_kb":644197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"1"}]},{"pgid":"2.2","version":"413'214142","reported_seq":365401,"reported_epoch":413,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:31.954412+0000","last_change":"2023-10-03T18:29:31.242675+0000","last_active":"2023-10-03T18:29:31.954412+0000","last_peered":"2023-10-03T18:29:31.954412+0000","last_clean":"2023-10-03T18:29:31.954412+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:29:31.954412+0000","last_undegraded":"2023-10-03T18:29:31.954412+0000","last_fullsized":"2023-10-03T18:29:31.954412+0000","mapping_epoch":12,"log_start":"371'204123","ondisk_log_start":"371'204123","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:37:30.772170+0000","scrub_duration":370.574046178,"objects_trimmed":65,"snaptrim_duration":3.6002977540000001,"stat_sum":{"num_bytes":2898707256,"num_objects":3080,"num_object_clones":2437,"num_object_copies":6160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3080,"num_whiteouts":0,"num_read":190854,"num_read_kb":657961,"num_write":162974,"num_write_kb":648214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b3","length":"5"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"3"}]},{"pgid":"2.4","version":"414'198348","reported_seq":346752,"reported_epoch":414,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:32.792999+0000","last_change":"2023-10-03T18:29:31.512234+0000","last_active":"2023-10-03T18:29:32.792999+0000","last_peered":"2023-10-03T18:29:32.792999+0000","last_clean":"2023-10-03T18:29:32.792999+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:29:32.792999+0000","last_undegraded":"2023-10-03T18:29:32.792999+0000","last_fullsized":"2023-10-03T18:29:32.792999+0000","mapping_epoch":12,"log_start":"365'188266","ondisk_log_start":"365'188266","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10082,"log_dups_size":0,"ondisk_log_size":10082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":236,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.42434037299999999,"stat_sum":{"num_bytes":2884698750,"num_objects":9865,"num_object_clones":9229,"num_object_copies":19730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9865,"num_whiteouts":0,"num_read":176572,"num_read_kb":642371,"num_write":160658,"num_write_kb":640572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b4","length":"4"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23958570257,"num_objects":70109,"num_object_clones":64845,"num_object_copies":140218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70109,"num_whiteouts":0,"num_read":1481711,"num_read_kb":5356300,"num_write":1338072,"num_write_kb":5332605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10912280576,"data_stored":127690160674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80387,"ondisk_log_size":80387,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739050,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3628976,"kb_used_data":2050536,"kb_used_omap":1375,"kb_used_meta":1577056,"kb_avail":90742864,"statfs":{"total":96636764160,"available":92920692736,"internally_reserved":0,"allocated":2099748864,"data_stored":21247845173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1614905476},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,0,1,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":2.379,"15min":2.4929999999999999},"min":{"1min":0.221,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":9.2859999999999996,"5min":14.26,"15min":18.321000000000002},"last":0.307},{"interface":"front","average":{"1min":2.5099999999999998,"5min":2.879,"15min":2.4769999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.4120000000000008,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.33000000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8939999999999999,"5min":2.698,"15min":2.456},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.3319999999999999,"5min":15.85,"15min":17.274000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":2.2850000000000001,"5min":2.5030000000000001,"15min":2.2709999999999999},"min":{"1min":0.245,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.3719999999999999,"5min":13.842000000000001,"15min":13.842000000000001},"last":1.3420000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739048,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4334096,"kb_used_data":3286792,"kb_used_omap":2774,"kb_used_meta":1044521,"kb_avail":90037744,"statfs":{"total":96636764160,"available":92198649856,"internally_reserved":0,"allocated":3365675008,"data_stored":42602549998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1069590111},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,0,0,1,1,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:28:47 2023","interfaces":[{"interface":"back","average":{"1min":4.3639999999999999,"5min":4.1399999999999997,"15min":3.202},"min":{"1min":0.22700000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":19.873000000000001,"5min":45.886000000000003,"15min":45.886000000000003},"last":11.481999999999999},{"interface":"front","average":{"1min":4.0439999999999996,"5min":3.6139999999999999,"15min":3.3279999999999998},"min":{"1min":0.26600000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":12.581,"5min":45.555,"15min":45.555},"last":12.815}]},{"osd":2,"last update":"Tue Oct 3 18:28:47 2023","interfaces":[{"interface":"back","average":{"1min":1.944,"5min":2.1099999999999999,"15min":2},"min":{"1min":0.32500000000000001,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":7.7880000000000003,"5min":11.252000000000001,"15min":13.209},"last":0.29499999999999998},{"interface":"front","average":{"1min":3.254,"5min":3.1960000000000002,"15min":2.6040000000000001},"min":{"1min":0.26500000000000001,"5min":0.192,"15min":0.192},"max":{"1min":12.563000000000001,"5min":25.77,"15min":25.77},"last":5.7720000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739019,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7312740,"kb_used_data":5332952,"kb_used_omap":4927,"kb_used_meta":1974848,"kb_avail":87059100,"statfs":{"total":96636764160,"available":89148518400,"internally_reserved":0,"allocated":5460942848,"data_stored":63870247354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":2022244813},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,3,10,8,17,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":2.5259999999999998,"5min":3.2519999999999998,"15min":3.2989999999999999},"min":{"1min":0.32100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":8.8369999999999997,"5min":20.466999999999999,"15min":30.896999999999998},"last":1.7},{"interface":"front","average":{"1min":1.3640000000000001,"5min":2.9729999999999999,"15min":2.8969999999999998},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.17799999999999999},"max":{"1min":4.5510000000000002,"5min":24.917000000000002,"15min":24.917000000000002},"last":1.7270000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:28:38 2023","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":2.8959999999999999,"15min":2.2789999999999999},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.9279999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.34599999999999997},{"interface":"front","average":{"1min":1.827,"5min":2.9180000000000001,"15min":2.8479999999999999},"min":{"1min":0.253,"5min":0.186,"15min":0.186},"max":{"1min":4.9429999999999996,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.498}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3360956416,"data_stored":42599930981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5455896576,"data_stored":63851689233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2095427584,"data_stored":21238540460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:29:35.561 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:35.561 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:35.562 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:35.562 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:35.562 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:29:36.593 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.592+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:36.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.598+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:36.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.607+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:36.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.709+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:36.716 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.709+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:36.732 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.731+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:36.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.806+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:37.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.827+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:37.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.846+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:37.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:36.908+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:39.544 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.542+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:39.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.572+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:39.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.745+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:39.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.794+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:39.801 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.794+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:39.807 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.806+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:39.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.887+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:39.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.895+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:39.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:39.946+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:40.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.652+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:40.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.669+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:40.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.669+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:40.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.669+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:40.712 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.711+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:40.791 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.790+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:40.817 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:40.816+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:41.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:41.847+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:29:41.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:41.989+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:41.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:41.989+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:41.991 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:41.989+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:42.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.123+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:42.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.123+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:42.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.219+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:42.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.219+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:42.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.267+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:42.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.327+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:42.369 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.363+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:42.915 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.914+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:42.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.963+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:42.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:42.995+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:43.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.048+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:43.106 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.105+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:43.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.167+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:43.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.190+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:43.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.190+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:43.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.190+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:43.192 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:43.191+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:44.498 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.498+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:44.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.502+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:44.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.502+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:44.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.517+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:44.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.523+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:44.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:44.534+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:49.889 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.887+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:49.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.929+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:49.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.929+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:49.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.929+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:49.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.929+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:49.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.949+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:49.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.949+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:49.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.949+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:49.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.949+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:49.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:49.949+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:52.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.663+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:52.664 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.663+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:52.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.672+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:52.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.672+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:52.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.672+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:52.675 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.672+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:52.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.802+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:52.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.815+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:29:52.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.829+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:29:52.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.858+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:52.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.858+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:52.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.895+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:52.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.916+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:52.928 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.916+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:52.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.928+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:52.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:52.928+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:53.294 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.287+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:29:53.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.334+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:53.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.353+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:53.370 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.370+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:53.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.384+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:53.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.384+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:53.385 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.384+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:53.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.397+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:53.446 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.441+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:53.904 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.903+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:53.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.941+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:53.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.941+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:53.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.961+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:53.964 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.963+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:53.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:53.984+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:55.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:29:56.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:29:56.478 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:29:56.510 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2123,"stamp":"2023-10-03T18:29:55.444674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":23984957233,"num_objects":69807,"num_object_clones":64541,"num_object_copies":139614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69807,"num_whiteouts":0,"num_read":1488895,"num_read_kb":5382208,"num_write":1344492,"num_write_kb":5359631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80473,"ondisk_log_size":80473,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15578232,"kb_used_data":10718688,"kb_used_omap":9077,"kb_used_meta":4850442,"kb_avail":267537288,"statfs":{"total":289910292480,"available":273958182912,"internally_reserved":0,"allocated":10975936512,"data_stored":127320406237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":4966852784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,2,2,9,13,14,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12705792,"num_objects":-104,"num_object_clones":-104,"num_object_copies":-208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-104,"num_whiteouts":0,"num_read":3910,"num_read_kb":13169,"num_write":3093,"num_write_kb":12372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001595"},"pg_stats":[{"pgid":"2.7","version":"417'204332","reported_seq":357874,"reported_epoch":417,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:54.483411+0000","last_change":"2023-10-03T18:29:52.398997+0000","last_active":"2023-10-03T18:29:54.483411+0000","last_peered":"2023-10-03T18:29:54.483411+0000","last_clean":"2023-10-03T18:29:54.483411+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:29:54.483411+0000","last_undegraded":"2023-10-03T18:29:54.483411+0000","last_fullsized":"2023-10-03T18:29:54.483411+0000","mapping_epoch":12,"log_start":"370'194288","ondisk_log_start":"370'194288","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":310,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:29:13.880240+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":4.7225542420000002,"stat_sum":{"num_bytes":2987835827,"num_objects":12603,"num_object_clones":11953,"num_object_copies":25206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12603,"num_whiteouts":0,"num_read":181495,"num_read_kb":669792,"num_write":167042,"num_write_kb":666144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b7","length":"1"}]},{"pgid":"2.6","version":"417'189035","reported_seq":331936,"reported_epoch":417,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:54.478488+0000","last_change":"2023-10-03T18:29:51.850575+0000","last_active":"2023-10-03T18:29:54.478488+0000","last_peered":"2023-10-03T18:29:54.478488+0000","last_clean":"2023-10-03T18:29:54.478488+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:29:54.478488+0000","last_undegraded":"2023-10-03T18:29:54.478488+0000","last_fullsized":"2023-10-03T18:29:54.478488+0000","mapping_epoch":12,"log_start":"367'178964","ondisk_log_start":"367'178964","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":225,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.193501479,"stat_sum":{"num_bytes":2760634991,"num_objects":9256,"num_object_clones":8652,"num_object_copies":18512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9256,"num_whiteouts":0,"num_read":168349,"num_read_kb":618196,"num_write":154340,"num_write_kb":616005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b7","length":"1"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"3"}]},{"pgid":"2.5","version":"417'226791","reported_seq":395196,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:54.498016+0000","last_change":"2023-10-03T18:29:54.305340+0000","last_active":"2023-10-03T18:29:54.498016+0000","last_peered":"2023-10-03T18:29:54.498016+0000","last_clean":"2023-10-03T18:29:54.498016+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:29:54.498016+0000","last_undegraded":"2023-10-03T18:29:54.498016+0000","last_fullsized":"2023-10-03T18:29:54.498016+0000","mapping_epoch":12,"log_start":"379'216770","ondisk_log_start":"379'216770","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":259,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.5114564550000003,"stat_sum":{"num_bytes":3238474381,"num_objects":11960,"num_object_clones":11250,"num_object_copies":23920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11960,"num_whiteouts":0,"num_read":207902,"num_read_kb":731874,"num_write":182057,"num_write_kb":724545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b7","length":"1"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"1"}]},{"pgid":"2.0","version":"416'223086","reported_seq":381224,"reported_epoch":416,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:50.400454+0000","last_change":"2023-10-03T18:29:48.275173+0000","last_active":"2023-10-03T18:29:50.400454+0000","last_peered":"2023-10-03T18:29:50.400454+0000","last_clean":"2023-10-03T18:29:50.400454+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:29:50.400454+0000","last_undegraded":"2023-10-03T18:29:50.400454+0000","last_fullsized":"2023-10-03T18:29:50.400454+0000","mapping_epoch":12,"log_start":"383'213064","ondisk_log_start":"383'213064","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10022,"log_dups_size":0,"ondisk_log_size":10022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":172,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:06:36.476597+0000","scrub_duration":734.24538164700004,"objects_trimmed":39,"snaptrim_duration":0.60408463199999995,"stat_sum":{"num_bytes":3127767846,"num_objects":8732,"num_object_clones":8044,"num_object_copies":17464,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8732,"num_whiteouts":0,"num_read":188367,"num_read_kb":696732,"num_write":175419,"num_write_kb":699656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b6","length":"2"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":582,"reported_epoch":416,"state":"active+clean","last_fresh":"2023-10-03T18:29:47.667385+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:29:47.667385+0000","last_peered":"2023-10-03T18:29:47.667385+0000","last_clean":"2023-10-03T18:29:47.667385+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:29:47.667385+0000","last_undegraded":"2023-10-03T18:29:47.667385+0000","last_fullsized":"2023-10-03T18:29:47.667385+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"417'218255","reported_seq":379785,"reported_epoch":417,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:29:54.362378+0000","last_change":"2023-10-03T18:29:54.288414+0000","last_active":"2023-10-03T18:29:54.362378+0000","last_peered":"2023-10-03T18:29:54.362378+0000","last_clean":"2023-10-03T18:29:54.362378+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:29:54.362378+0000","last_undegraded":"2023-10-03T18:29:54.362378+0000","last_fullsized":"2023-10-03T18:29:54.362378+0000","mapping_epoch":12,"log_start":"372'208177","ondisk_log_start":"372'208177","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":249,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":1.5944950690000002,"stat_sum":{"num_bytes":3181622141,"num_objects":11459,"num_object_clones":10759,"num_object_copies":22918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11459,"num_whiteouts":0,"num_read":192146,"num_read_kb":709909,"num_write":177992,"num_write_kb":709860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b7","length":"1"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"1"}]},{"pgid":"2.1","version":"416'215162","reported_seq":361455,"reported_epoch":416,"state":"active+clean","last_fresh":"2023-10-03T18:29:50.486102+0000","last_change":"2023-10-03T18:29:42.477821+0000","last_active":"2023-10-03T18:29:50.486102+0000","last_peered":"2023-10-03T18:29:50.486102+0000","last_clean":"2023-10-03T18:29:50.486102+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:29:50.486102+0000","last_undegraded":"2023-10-03T18:29:50.486102+0000","last_fullsized":"2023-10-03T18:29:50.486102+0000","mapping_epoch":12,"log_start":"380'205086","ondisk_log_start":"380'205086","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:11:34.357820+0000","scrub_duration":424.08306342600002,"objects_trimmed":164,"snaptrim_duration":9.8333177509999992,"stat_sum":{"num_bytes":2898240009,"num_objects":2966,"num_object_clones":2333,"num_object_copies":5932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2966,"num_whiteouts":0,"num_read":181223,"num_read_kb":648604,"num_write":162320,"num_write_kb":646529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1b6","length":"2"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"}]},{"pgid":"2.2","version":"416'215362","reported_seq":367303,"reported_epoch":416,"state":"active+clean","last_fresh":"2023-10-03T18:29:52.242278+0000","last_change":"2023-10-03T18:29:44.039389+0000","last_active":"2023-10-03T18:29:52.242278+0000","last_peered":"2023-10-03T18:29:52.242278+0000","last_clean":"2023-10-03T18:29:52.242278+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:29:52.242278+0000","last_undegraded":"2023-10-03T18:29:52.242278+0000","last_fullsized":"2023-10-03T18:29:52.242278+0000","mapping_epoch":12,"log_start":"380'205327","ondisk_log_start":"380'205327","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:37:30.772170+0000","scrub_duration":370.574046178,"objects_trimmed":23,"snaptrim_duration":1.3979464799999999,"stat_sum":{"num_bytes":2901574456,"num_objects":2968,"num_object_clones":2325,"num_object_copies":5936,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2968,"num_whiteouts":0,"num_read":191778,"num_read_kb":661258,"num_write":163675,"num_write_kb":651018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b6","length":"2"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"}]},{"pgid":"2.4","version":"417'199599","reported_seq":348706,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:29:54.492960+0000","last_change":"2023-10-03T18:29:52.429390+0000","last_active":"2023-10-03T18:29:54.492960+0000","last_peered":"2023-10-03T18:29:54.492960+0000","last_clean":"2023-10-03T18:29:54.492960+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:29:54.492960+0000","last_undegraded":"2023-10-03T18:29:54.492960+0000","last_fullsized":"2023-10-03T18:29:54.492960+0000","mapping_epoch":12,"log_start":"369'189572","ondisk_log_start":"369'189572","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":232,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":22,"snaptrim_duration":2.3847374449999998,"stat_sum":{"num_bytes":2888217214,"num_objects":9861,"num_object_clones":9225,"num_object_copies":19722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9861,"num_whiteouts":0,"num_read":177529,"num_read_kb":645752,"num_write":161510,"num_write_kb":643980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b7","length":"1"},{"start":"1bc","length":"2"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"8"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":23984366865,"num_objects":69805,"num_object_clones":64541,"num_object_copies":139610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69805,"num_whiteouts":0,"num_read":1488789,"num_read_kb":5382117,"num_write":1344355,"num_write_kb":5357737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10963738624,"data_stored":127314508322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80374,"ondisk_log_size":80374,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739053,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3664292,"kb_used_data":2058076,"kb_used_omap":1375,"kb_used_meta":1604832,"kb_avail":90707548,"statfs":{"total":96636764160,"available":92884529152,"internally_reserved":0,"allocated":2107469824,"data_stored":21077303157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1643348100},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,3,2,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":2.379,"15min":2.4929999999999999},"min":{"1min":0.221,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":9.2859999999999996,"5min":14.26,"15min":18.321000000000002},"last":0.35199999999999998},{"interface":"front","average":{"1min":2.5099999999999998,"5min":2.879,"15min":2.4769999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.4120000000000008,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.27000000000000002}]},{"osd":1,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8939999999999999,"5min":2.698,"15min":2.456},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.3319999999999999,"5min":15.85,"15min":17.274000000000001},"last":4.2370000000000001},{"interface":"front","average":{"1min":2.2850000000000001,"5min":2.5030000000000001,"15min":2.2709999999999999},"min":{"1min":0.245,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.3719999999999999,"5min":13.842000000000001,"15min":13.842000000000001},"last":4.2169999999999996}]}]},{"osd":0,"up_from":8,"seq":34359739052,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4417072,"kb_used_data":3301736,"kb_used_omap":2774,"kb_used_meta":1112553,"kb_avail":89954768,"statfs":{"total":96636764160,"available":92113682432,"internally_reserved":0,"allocated":3380977664,"data_stored":42601555278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1139254879},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":3.9620000000000002,"15min":3.177},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":11.481999999999999,"5min":45.886000000000003,"15min":45.886000000000003},"last":4.3810000000000002},{"interface":"front","average":{"1min":4.71,"5min":4.0899999999999999,"15min":3.403},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":14.523,"5min":45.555,"15min":45.555},"last":12.92}]},{"osd":2,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.1339999999999999,"5min":2.2850000000000001,"15min":2.0070000000000001},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":6.8419999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.194,"5min":3.2440000000000002,"15min":2.5299999999999998},"min":{"1min":0.29199999999999998,"5min":0.192,"15min":0.192},"max":{"1min":5.7720000000000002,"5min":25.77,"15min":25.77},"last":11.609}]}]},{"osd":1,"up_from":8,"seq":34359739023,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7496868,"kb_used_data":5358876,"kb_used_omap":4927,"kb_used_meta":2133056,"kb_avail":86874972,"statfs":{"total":96636764160,"available":88959971328,"internally_reserved":0,"allocated":5487489024,"data_stored":63641547802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":2184249805},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,4,11,9,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":7.2190000000000003,"5min":4.1029999999999998,"15min":3.6440000000000001},"min":{"1min":0.35499999999999998,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":53.280000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":10.397},{"interface":"front","average":{"1min":5.0759999999999996,"5min":3.3769999999999998,"15min":3.0670000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":52.982999999999997,"5min":52.982999999999997,"15min":52.982999999999997},"last":10.339}]},{"osd":2,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.8250000000000002,"15min":2.3540000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.436999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.3290000000000002,"5min":2.7120000000000002,"15min":2.8140000000000001},"min":{"1min":0.23499999999999999,"5min":0.215,"15min":0.186},"max":{"1min":12.286,"5min":36.332999999999998,"15min":36.332999999999998},"last":1.248}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3376594944,"data_stored":42598079589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5483896832,"data_stored":63639475473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2103246848,"data_stored":21076953260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:29:56.512 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:56.512 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:56.512 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:56.512 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:29:56.513 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:29:57.633 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:57.624+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:57.634 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:57.624+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:57.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:57.632+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:57.649 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:57.640+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:29:59.500 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.493+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:29:59.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.547+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:29:59.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.547+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:29:59.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.555+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:29:59.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.555+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:29:59.560 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.555+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:29:59.561 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.560+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:29:59.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:29:59.564+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:00.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.338+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:00.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.346+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:00.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.347+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:00.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.347+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:00.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.410+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:00.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.421+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:00.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.421+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:00.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.421+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:00.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.421+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:00.422 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.421+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:00.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.694+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:00.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.694+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:00.722 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.720+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:00.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.726+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:00.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.726+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:00.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.757+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:00.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.757+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:00.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.757+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:00.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.757+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:00.758 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:00.757+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:00.972 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:30:00.966+0000 7f9f21f20700 -1 reset not still connected to 0x5632f3f29520 2023-10-03T18:30:01.094 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:01.089+0000 7fc515f98700 -1 reset not still connected to 0x5622077dd040 2023-10-03T18:30:01.113 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:01.113+0000 7fc515f98700 -1 reset not still connected to 0x56221642dee0 2023-10-03T18:30:01.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:01.231+0000 7fc515f98700 -1 reset not still connected to 0x56221c5fbba0 2023-10-03T18:30:01.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:01.231+0000 7fc515f98700 -1 reset not still connected to 0x56222374dc70 2023-10-03T18:30:04.582 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:04.580+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:04.756 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:04.753+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:04.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:04.863+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:04.942 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:04.935+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:05.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:05.005+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:05.200 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:05.198+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:05.240 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:05.237+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:05.240 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:05.237+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:05.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:05.875+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:06.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:06.013+0000 7fc515f98700 -1 reset not still connected to 0x56220f6904e0 2023-10-03T18:30:06.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:06.013+0000 7fc515f98700 -1 reset not still connected to 0x562234d64340 2023-10-03T18:30:09.204 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.200+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:09.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.252+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:09.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.458+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:09.505 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.504+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:09.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.546+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:09.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.575+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:09.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.575+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:09.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.575+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:09.608 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.605+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:09.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:09.767+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:09.837 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:30:09.830+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:30:09.887 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:30:09.880+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:30:09.888 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:30:09.880+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:30:09.888 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:30:09.880+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:30:11.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.352+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:11.473 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.472+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:11.483 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.473+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:11.539 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.535+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:11.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.630+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:11.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.630+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:11.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.686+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:11.687 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.686+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:11.852 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.852+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:11.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.861+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:11.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:11.930+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:12.381 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.378+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:12.448 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.447+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:12.527 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.526+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:12.542 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.534+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:12.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.555+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:12.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.565+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:12.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.595+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:12.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.640+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:12.678 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.676+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:12.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:12.695+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:13.447 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.446+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:13.452 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.451+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:13.555 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.554+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:13.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.587+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:13.591 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.587+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:13.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.642+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:13.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.642+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:13.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:13.642+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:16.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:30:17.169 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:30:17.176 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:30:17.193 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2142,"stamp":"2023-10-03T18:30:15.463309+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24005343025,"num_objects":70480,"num_object_clones":65214,"num_object_copies":140960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70480,"num_whiteouts":0,"num_read":1497868,"num_read_kb":5406238,"num_write":1349672,"num_write_kb":5379631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80574,"ondisk_log_size":80574,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15945500,"kb_used_data":10761976,"kb_used_omap":9077,"kb_used_meta":5174410,"kb_avail":267170020,"statfs":{"total":289910292480,"available":273582100480,"internally_reserved":0,"allocated":11020263424,"data_stored":129064888551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":5298596016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,5,2,8,12,23,14,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4878183,"num_objects":567,"num_object_clones":570,"num_object_copies":1134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":1593,"num_read_kb":5278,"num_write":1232,"num_write_kb":4800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.659010"},"pg_stats":[{"pgid":"2.7","version":"425'205269","reported_seq":359285,"reported_epoch":425,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:11.343487+0000","last_change":"2023-10-03T18:30:11.311830+0000","last_active":"2023-10-03T18:30:11.343487+0000","last_peered":"2023-10-03T18:30:11.343487+0000","last_clean":"2023-10-03T18:30:11.343487+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:30:11.343487+0000","last_undegraded":"2023-10-03T18:30:11.343487+0000","last_fullsized":"2023-10-03T18:30:11.343487+0000","mapping_epoch":12,"log_start":"379'195192","ondisk_log_start":"379'195192","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10077,"log_dups_size":0,"ondisk_log_size":10077,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":328,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:37:33.952265+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.029410355999999999,"stat_sum":{"num_bytes":2990219699,"num_objects":12748,"num_object_clones":12098,"num_object_copies":25496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12748,"num_whiteouts":0,"num_read":182403,"num_read_kb":672441,"num_write":167639,"num_write_kb":668467,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1bd","length":"1"}]},{"pgid":"2.6","version":"425'189959","reported_seq":333372,"reported_epoch":425,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:11.267335+0000","last_change":"2023-10-03T18:30:07.527299+0000","last_active":"2023-10-03T18:30:11.267335+0000","last_peered":"2023-10-03T18:30:11.267335+0000","last_clean":"2023-10-03T18:30:11.267335+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:30:11.267335+0000","last_undegraded":"2023-10-03T18:30:11.267335+0000","last_fullsized":"2023-10-03T18:30:11.267335+0000","mapping_epoch":12,"log_start":"370'179866","ondisk_log_start":"370'179866","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10093,"log_dups_size":0,"ondisk_log_size":10093,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":242,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0.627280952,"stat_sum":{"num_bytes":2762859219,"num_objects":9361,"num_object_clones":8755,"num_object_copies":18722,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9361,"num_whiteouts":0,"num_read":169216,"num_read_kb":620745,"num_write":154894,"num_write_kb":618172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"1"}]},{"pgid":"2.5","version":"425'227858","reported_seq":396872,"reported_epoch":425,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:11.342487+0000","last_change":"2023-10-03T18:30:07.573349+0000","last_active":"2023-10-03T18:30:11.342487+0000","last_peered":"2023-10-03T18:30:11.342487+0000","last_clean":"2023-10-03T18:30:11.342487+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:30:11.342487+0000","last_undegraded":"2023-10-03T18:30:11.342487+0000","last_fullsized":"2023-10-03T18:30:11.342487+0000","mapping_epoch":12,"log_start":"381'217772","ondisk_log_start":"381'217772","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10086,"log_dups_size":0,"ondisk_log_size":10086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":277,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":0.19666244099999999,"stat_sum":{"num_bytes":3240972841,"num_objects":12089,"num_object_clones":11381,"num_object_copies":24178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12089,"num_whiteouts":0,"num_read":209239,"num_read_kb":735128,"num_write":182706,"num_write_kb":727003,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"3"}]},{"pgid":"2.0","version":"425'224707","reported_seq":383476,"reported_epoch":425,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:10.541904+0000","last_change":"2023-10-03T18:30:10.204268+0000","last_active":"2023-10-03T18:30:10.541904+0000","last_peered":"2023-10-03T18:30:10.541904+0000","last_clean":"2023-10-03T18:30:10.541904+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:30:10.541904+0000","last_undegraded":"2023-10-03T18:30:10.541904+0000","last_fullsized":"2023-10-03T18:30:10.541904+0000","mapping_epoch":12,"log_start":"387'214671","ondisk_log_start":"387'214671","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10036,"log_dups_size":0,"ondisk_log_size":10036,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":182,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:40:37.900562+0000","scrub_duration":734.24538164700004,"objects_trimmed":57,"snaptrim_duration":3.3845670019999998,"stat_sum":{"num_bytes":3130819317,"num_objects":8612,"num_object_clones":7925,"num_object_copies":17224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8612,"num_whiteouts":0,"num_read":189619,"num_read_kb":700402,"num_write":176190,"num_write_kb":702657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"}]},{"pgid":"1.0","version":"88'99","reported_seq":591,"reported_epoch":424,"state":"active+clean","last_fresh":"2023-10-03T18:30:05.914631+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:30:05.914631+0000","last_peered":"2023-10-03T18:30:05.914631+0000","last_clean":"2023-10-03T18:30:05.914631+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:30:05.914631+0000","last_undegraded":"2023-10-03T18:30:05.914631+0000","last_fullsized":"2023-10-03T18:30:05.914631+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"425'219228","reported_seq":381283,"reported_epoch":425,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:11.251197+0000","last_change":"2023-10-03T18:30:11.092407+0000","last_active":"2023-10-03T18:30:11.251197+0000","last_peered":"2023-10-03T18:30:11.251197+0000","last_clean":"2023-10-03T18:30:11.251197+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:30:11.251197+0000","last_undegraded":"2023-10-03T18:30:11.251197+0000","last_fullsized":"2023-10-03T18:30:11.251197+0000","mapping_epoch":12,"log_start":"380'209182","ondisk_log_start":"380'209182","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":266,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":41,"snaptrim_duration":4.2444730210000001,"stat_sum":{"num_bytes":3183948466,"num_objects":11519,"num_object_clones":10823,"num_object_copies":23038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11519,"num_whiteouts":0,"num_read":193083,"num_read_kb":712677,"num_write":178579,"num_write_kb":712134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"4"}]},{"pgid":"2.1","version":"425'216344","reported_seq":363238,"reported_epoch":425,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:10.569087+0000","last_change":"2023-10-03T18:30:09.319138+0000","last_active":"2023-10-03T18:30:10.569087+0000","last_peered":"2023-10-03T18:30:10.569087+0000","last_clean":"2023-10-03T18:30:10.569087+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:30:10.569087+0000","last_undegraded":"2023-10-03T18:30:10.569087+0000","last_fullsized":"2023-10-03T18:30:10.569087+0000","mapping_epoch":12,"log_start":"382'206289","ondisk_log_start":"382'206289","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:19:19.195819+0000","scrub_duration":424.08306342600002,"objects_trimmed":37,"snaptrim_duration":2.4799471469999999,"stat_sum":{"num_bytes":2901176842,"num_objects":3093,"num_object_clones":2460,"num_object_copies":6186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":182552,"num_read_kb":651916,"num_write":163065,"num_write_kb":649415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"3"}]},{"pgid":"2.2","version":"425'216566","reported_seq":369182,"reported_epoch":425,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:12.339867+0000","last_change":"2023-10-03T18:30:09.553651+0000","last_active":"2023-10-03T18:30:12.339867+0000","last_peered":"2023-10-03T18:30:12.339867+0000","last_clean":"2023-10-03T18:30:12.339867+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:30:12.339867+0000","last_undegraded":"2023-10-03T18:30:12.339867+0000","last_fullsized":"2023-10-03T18:30:12.339867+0000","mapping_epoch":12,"log_start":"383'206528","ondisk_log_start":"383'206528","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10038,"log_dups_size":0,"ondisk_log_size":10038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:06:25.748643+0000","scrub_duration":370.574046178,"objects_trimmed":72,"snaptrim_duration":2.7352426809999999,"stat_sum":{"num_bytes":2904228764,"num_objects":3058,"num_object_clones":2413,"num_object_copies":6116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3058,"num_whiteouts":0,"num_read":193265,"num_read_kb":664747,"num_write":164368,"num_write_kb":653647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"4"}]},{"pgid":"2.4","version":"425'200521","reported_seq":350061,"reported_epoch":425,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:11.333288+0000","last_change":"2023-10-03T18:30:08.757948+0000","last_active":"2023-10-03T18:30:11.333288+0000","last_peered":"2023-10-03T18:30:11.333288+0000","last_clean":"2023-10-03T18:30:11.333288+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:30:11.333288+0000","last_undegraded":"2023-10-03T18:30:11.333288+0000","last_fullsized":"2023-10-03T18:30:11.333288+0000","mapping_epoch":12,"log_start":"377'190477","ondisk_log_start":"377'190477","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":249,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":21,"snaptrim_duration":1.886449982,"stat_sum":{"num_bytes":2890527509,"num_objects":9998,"num_object_clones":9359,"num_object_copies":19996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9998,"num_whiteouts":0,"num_read":178385,"num_read_kb":648091,"num_write":162094,"num_write_kb":646242,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1bd","length":"1"},{"start":"1c4","length":"5"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24004752657,"num_objects":70478,"num_object_clones":65214,"num_object_copies":140956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70478,"num_whiteouts":0,"num_read":1497762,"num_read_kb":5406147,"num_write":1349535,"num_write_kb":5377737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11005308928,"data_stored":129115240994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80475,"ondisk_log_size":80475,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739057,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3735808,"kb_used_data":2067128,"kb_used_omap":1375,"kb_used_meta":1667296,"kb_avail":90636032,"statfs":{"total":96636764160,"available":92811296768,"internally_reserved":0,"allocated":2116739072,"data_stored":21190267016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1707311236},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,3,3,1,7,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.7310000000000001,"5min":2.379,"15min":2.4929999999999999},"min":{"1min":0.221,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":9.2859999999999996,"5min":14.26,"15min":18.321000000000002},"last":1.1890000000000001},{"interface":"front","average":{"1min":2.5099999999999998,"5min":2.879,"15min":2.4769999999999999},"min":{"1min":0.27100000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":8.4120000000000008,"5min":34.176000000000002,"15min":34.176000000000002},"last":1.345}]},{"osd":1,"last update":"Tue Oct 3 18:29:27 2023","interfaces":[{"interface":"back","average":{"1min":1.8939999999999999,"5min":2.698,"15min":2.456},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.3319999999999999,"5min":15.85,"15min":17.274000000000001},"last":5.726},{"interface":"front","average":{"1min":2.2850000000000001,"5min":2.5030000000000001,"15min":2.2709999999999999},"min":{"1min":0.245,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.3719999999999999,"5min":13.842000000000001,"15min":13.842000000000001},"last":5.6349999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739056,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4553420,"kb_used_data":3316352,"kb_used_omap":2774,"kb_used_meta":1234281,"kb_avail":89818420,"statfs":{"total":96636764160,"available":91974062080,"internally_reserved":0,"allocated":3395944448,"data_stored":43383900247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1263904351},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,1,2,15,0,6],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":3.9620000000000002,"15min":3.177},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":11.481999999999999,"5min":45.886000000000003,"15min":45.886000000000003},"last":1.131},{"interface":"front","average":{"1min":4.71,"5min":4.0899999999999999,"15min":3.403},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":14.523,"5min":45.555,"15min":45.555},"last":1.571}]},{"osd":2,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.1339999999999999,"5min":2.2850000000000001,"15min":2.0070000000000001},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":6.8419999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.186},{"interface":"front","average":{"1min":2.194,"5min":3.2440000000000002,"15min":2.5299999999999998},"min":{"1min":0.29199999999999998,"5min":0.192,"15min":0.192},"max":{"1min":5.7720000000000002,"5min":25.77,"15min":25.77},"last":0.22700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739026,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7656272,"kb_used_data":5378496,"kb_used_omap":4927,"kb_used_meta":2272832,"kb_avail":86715568,"statfs":{"total":96636764160,"available":88796741632,"internally_reserved":0,"allocated":5507579904,"data_stored":64490721288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":2327380429},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,0,4,7,7,7,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":7.2190000000000003,"5min":4.1029999999999998,"15min":3.6440000000000001},"min":{"1min":0.35499999999999998,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":53.280000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.89200000000000002},{"interface":"front","average":{"1min":5.0759999999999996,"5min":3.3769999999999998,"15min":3.0670000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":52.982999999999997,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.36299999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.8250000000000002,"15min":2.3540000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.436999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.51300000000000001},{"interface":"front","average":{"1min":2.3290000000000002,"5min":2.7120000000000002,"15min":2.8140000000000001},"min":{"1min":0.23499999999999999,"5min":0.215,"15min":0.186},"max":{"1min":12.286,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.32300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3390627840,"data_stored":43391393120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5502468096,"data_stored":64512541969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2112212992,"data_stored":21211305905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:30:17.195 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:17.195 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:17.196 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:17.196 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:17.196 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:30:17.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.686+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:17.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.792+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:17.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.792+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:17.800 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.798+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:17.845 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.842+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:17.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:17.851 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:17.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:20.070 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.061+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:20.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.108+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:20.112 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.108+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.132+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:20.134 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.132+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:20.285 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:20.281+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:24.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:24.881+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:24.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:24.910+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:24.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:24.910+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:24.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:24.932+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:24.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:24.939+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:25.029 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.024+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:25.086 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.083+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:25.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.122+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:25.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.181+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:25.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.181+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:25.223 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.222+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:25.493 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.491+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:25.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.633+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:25.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.633+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:25.642 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:25.640+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:26.138 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.137+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.192+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:26.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.259+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:26.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.272+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:26.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.272+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:26.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.358+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:26.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.390+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:26.425 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:26.424+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:27.240 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.238+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:27.264 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:27.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.387+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:27.393 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.387+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.548+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.548+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.548+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.549+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.549+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:27.553 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:27.549+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:29.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.042+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:29.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.107+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:29.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.107+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:29.109 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.107+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:29.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.117+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:29.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.128+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:29.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.128+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:29.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.160+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:29.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.167+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:29.229 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:29.227+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:32.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.852+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:32.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.898+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:32.907 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.902+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:32.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.928+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:32.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.928+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:32.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.928+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:32.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:32.935+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:33.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.046+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:33.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.555+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:33.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.555+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:33.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.555+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:33.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.702+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:33.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.702+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:33.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:33.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:33.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:33.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:33.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:33.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:33.720+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:34.415 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:34.413+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:35.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.539+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:35.541 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.539+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:35.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.559+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:35.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.575+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:35.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.575+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:35.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.575+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:35.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.680+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:35.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:35.680+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:37.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:30:38.137 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:30:38.148 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:30:38.166 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2156,"stamp":"2023-10-03T18:30:37.689219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24028170033,"num_objects":71349,"num_object_clones":66083,"num_object_copies":142698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71349,"num_whiteouts":0,"num_read":1503660,"num_read_kb":5429314,"num_write":1355218,"num_write_kb":5401815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80499,"ondisk_log_size":80499,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16276612,"kb_used_data":10807400,"kb_used_omap":9077,"kb_used_meta":5460106,"kb_avail":266838908,"statfs":{"total":289910292480,"available":273243041792,"internally_reserved":0,"allocated":11066777600,"data_stored":131437585111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":5591148720},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,6,14,12,8,22,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7659520,"num_objects":-76,"num_object_clones":-76,"num_object_copies":-152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-76,"num_whiteouts":0,"num_read":1919,"num_read_kb":7652,"num_write":1856,"num_write_kb":7424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.220722"},"pg_stats":[{"pgid":"2.7","version":"428'206454","reported_seq":360845,"reported_epoch":428,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:33.507308+0000","last_change":"2023-10-03T18:30:32.942208+0000","last_active":"2023-10-03T18:30:33.507308+0000","last_peered":"2023-10-03T18:30:33.507308+0000","last_clean":"2023-10-03T18:30:33.507308+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:30:33.507308+0000","last_undegraded":"2023-10-03T18:30:33.507308+0000","last_fullsized":"2023-10-03T18:30:33.507308+0000","mapping_epoch":12,"log_start":"383'196396","ondisk_log_start":"383'196396","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10058,"log_dups_size":0,"ondisk_log_size":10058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":325,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:37:33.952265+0000","scrub_duration":505.38444223300002,"objects_trimmed":4,"snaptrim_duration":5.3576955890000004,"stat_sum":{"num_bytes":2992955827,"num_objects":12947,"num_object_clones":12297,"num_object_copies":25894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12947,"num_whiteouts":0,"num_read":183107,"num_read_kb":675253,"num_write":168300,"num_write_kb":671111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c6","length":"1"}]},{"pgid":"2.6","version":"428'191087","reported_seq":334794,"reported_epoch":428,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:33.371019+0000","last_change":"2023-10-03T18:30:31.646334+0000","last_active":"2023-10-03T18:30:33.371019+0000","last_peered":"2023-10-03T18:30:33.371019+0000","last_clean":"2023-10-03T18:30:33.371019+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:30:33.371019+0000","last_undegraded":"2023-10-03T18:30:33.371019+0000","last_fullsized":"2023-10-03T18:30:33.371019+0000","mapping_epoch":12,"log_start":"380'181069","ondisk_log_start":"380'181069","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":239,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":5.106690918,"stat_sum":{"num_bytes":2765259475,"num_objects":9507,"num_object_clones":8901,"num_object_copies":19014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9507,"num_whiteouts":0,"num_read":169827,"num_read_kb":623181,"num_write":155481,"num_write_kb":620520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"1"}]},{"pgid":"2.5","version":"428'229129","reported_seq":398608,"reported_epoch":428,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:33.382673+0000","last_change":"2023-10-03T18:30:31.636720+0000","last_active":"2023-10-03T18:30:33.382673+0000","last_peered":"2023-10-03T18:30:33.382673+0000","last_clean":"2023-10-03T18:30:33.382673+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:30:33.382673+0000","last_undegraded":"2023-10-03T18:30:33.382673+0000","last_fullsized":"2023-10-03T18:30:33.382673+0000","mapping_epoch":12,"log_start":"384'219074","ondisk_log_start":"384'219074","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":273,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":8.9139773659999992,"stat_sum":{"num_bytes":3243823657,"num_objects":12316,"num_object_clones":11608,"num_object_copies":24632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12316,"num_whiteouts":0,"num_read":210047,"num_read_kb":738344,"num_write":183398,"num_write_kb":729771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"7"}]},{"pgid":"2.0","version":"428'226567","reported_seq":385652,"reported_epoch":428,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:35.718969+0000","last_change":"2023-10-03T18:30:34.627680+0000","last_active":"2023-10-03T18:30:35.718969+0000","last_peered":"2023-10-03T18:30:35.718969+0000","last_clean":"2023-10-03T18:30:35.718969+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:30:35.718969+0000","last_undegraded":"2023-10-03T18:30:35.718969+0000","last_fullsized":"2023-10-03T18:30:35.718969+0000","mapping_epoch":12,"log_start":"397'216486","ondisk_log_start":"397'216486","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":171,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:40:37.900562+0000","scrub_duration":734.24538164700004,"objects_trimmed":40,"snaptrim_duration":1.903505172,"stat_sum":{"num_bytes":3134173941,"num_objects":8628,"num_object_clones":7941,"num_object_copies":17256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8628,"num_whiteouts":0,"num_read":190412,"num_read_kb":703566,"num_write":177004,"num_write_kb":705913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":594,"reported_epoch":427,"state":"active+clean","last_fresh":"2023-10-03T18:30:22.759387+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:30:22.759387+0000","last_peered":"2023-10-03T18:30:22.759387+0000","last_clean":"2023-10-03T18:30:22.759387+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:30:22.759387+0000","last_undegraded":"2023-10-03T18:30:22.759387+0000","last_fullsized":"2023-10-03T18:30:22.759387+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"428'220442","reported_seq":382877,"reported_epoch":428,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:33.331587+0000","last_change":"2023-10-03T18:30:32.942209+0000","last_active":"2023-10-03T18:30:33.331587+0000","last_peered":"2023-10-03T18:30:33.331587+0000","last_clean":"2023-10-03T18:30:33.331587+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:30:33.331587+0000","last_undegraded":"2023-10-03T18:30:33.331587+0000","last_fullsized":"2023-10-03T18:30:33.331587+0000","mapping_epoch":12,"log_start":"383'210389","ondisk_log_start":"383'210389","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10053,"log_dups_size":0,"ondisk_log_size":10053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":263,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.228251185,"stat_sum":{"num_bytes":3186696882,"num_objects":11748,"num_object_clones":11052,"num_object_copies":23496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11748,"num_whiteouts":0,"num_read":193816,"num_read_kb":715593,"num_write":179242,"num_write_kb":714786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"7"}]},{"pgid":"2.1","version":"428'218171","reported_seq":365376,"reported_epoch":428,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:35.798989+0000","last_change":"2023-10-03T18:30:35.752722+0000","last_active":"2023-10-03T18:30:35.798989+0000","last_peered":"2023-10-03T18:30:35.798989+0000","last_clean":"2023-10-03T18:30:35.798989+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:30:35.798989+0000","last_undegraded":"2023-10-03T18:30:35.798989+0000","last_fullsized":"2023-10-03T18:30:35.798989+0000","mapping_epoch":12,"log_start":"390'208096","ondisk_log_start":"390'208096","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:19:19.195819+0000","scrub_duration":424.08306342600002,"objects_trimmed":56,"snaptrim_duration":3.056053575,"stat_sum":{"num_bytes":2904232458,"num_objects":2992,"num_object_clones":2359,"num_object_copies":5984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":183273,"num_read_kb":654780,"num_write":163811,"num_write_kb":652399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"1"}]},{"pgid":"2.2","version":"428'218362","reported_seq":371352,"reported_epoch":428,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:37.506930+0000","last_change":"2023-10-03T18:30:35.394942+0000","last_active":"2023-10-03T18:30:37.506930+0000","last_peered":"2023-10-03T18:30:37.506930+0000","last_clean":"2023-10-03T18:30:37.506930+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:30:37.506930+0000","last_undegraded":"2023-10-03T18:30:37.506930+0000","last_fullsized":"2023-10-03T18:30:37.506930+0000","mapping_epoch":12,"log_start":"392'208348","ondisk_log_start":"392'208348","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:06:25.748643+0000","scrub_duration":370.574046178,"objects_trimmed":73,"snaptrim_duration":2.6231903989999998,"stat_sum":{"num_bytes":2907259804,"num_objects":3003,"num_object_clones":2358,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3003,"num_whiteouts":0,"num_read":194040,"num_read_kb":667835,"num_write":165108,"num_write_kb":656607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"1"}]},{"pgid":"2.4","version":"428'201625","reported_seq":351515,"reported_epoch":428,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:33.358436+0000","last_change":"2023-10-03T18:30:30.237590+0000","last_active":"2023-10-03T18:30:33.358436+0000","last_peered":"2023-10-03T18:30:33.358436+0000","last_clean":"2023-10-03T18:30:33.358436+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:30:33.358436+0000","last_undegraded":"2023-10-03T18:30:33.358436+0000","last_fullsized":"2023-10-03T18:30:33.358436+0000","mapping_epoch":12,"log_start":"381'191579","ondisk_log_start":"381'191579","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10046,"log_dups_size":0,"ondisk_log_size":10046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":246,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":7.4659517409999996,"stat_sum":{"num_bytes":2893177621,"num_objects":10206,"num_object_clones":9567,"num_object_copies":20412,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10206,"num_whiteouts":0,"num_read":179032,"num_read_kb":650671,"num_write":162737,"num_write_kb":648814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1c6","length":"3"},{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"2"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24027579665,"num_objects":71347,"num_object_clones":66083,"num_object_copies":142694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71347,"num_whiteouts":0,"num_read":1503554,"num_read_kb":5429223,"num_write":1355081,"num_write_kb":5399921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11051843584,"data_stored":131438610978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80400,"ondisk_log_size":80400,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739062,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3795284,"kb_used_data":2076620,"kb_used_omap":1375,"kb_used_meta":1717280,"kb_avail":90576556,"statfs":{"total":96636764160,"available":92750393344,"internally_reserved":0,"allocated":2126458880,"data_stored":21476152280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1758494852},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,5,4,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.0840000000000001,"15min":2.298},"min":{"1min":0.17199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.401999999999999,"5min":14.26,"15min":15.964},"last":9.0340000000000007},{"interface":"front","average":{"1min":1.431,"5min":2.5230000000000001,"15min":2.3559999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.6699999999999999,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.374}]},{"osd":1,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":2.6899999999999999,"15min":2.4089999999999998},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":9.3219999999999992,"5min":15.85,"15min":17.274000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":2.2080000000000002,"5min":2.3420000000000001,"15min":2.2559999999999998},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.1479999999999997,"5min":13.842000000000001,"15min":13.842000000000001},"last":3.0459999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739061,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4636432,"kb_used_data":3331336,"kb_used_omap":2774,"kb_used_meta":1302313,"kb_avail":89735408,"statfs":{"total":96636764160,"available":91889057792,"internally_reserved":0,"allocated":3411288064,"data_stored":44243643303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1333569119},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.6699999999999999,"5min":3.9620000000000002,"15min":3.177},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":11.481999999999999,"5min":45.886000000000003,"15min":45.886000000000003},"last":0.29499999999999998},{"interface":"front","average":{"1min":4.71,"5min":4.0899999999999999,"15min":3.403},"min":{"1min":0.25700000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":14.523,"5min":45.555,"15min":45.555},"last":20.763000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:29:47 2023","interfaces":[{"interface":"back","average":{"1min":2.1339999999999999,"5min":2.2850000000000001,"15min":2.0070000000000001},"min":{"1min":0.27200000000000002,"5min":0.19600000000000001,"15min":0.192},"max":{"1min":6.8419999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.23300000000000001},{"interface":"front","average":{"1min":2.194,"5min":3.2440000000000002,"15min":2.5299999999999998},"min":{"1min":0.29199999999999998,"5min":0.192,"15min":0.192},"max":{"1min":5.7720000000000002,"5min":25.77,"15min":25.77},"last":18.352}]}]},{"osd":1,"up_from":8,"seq":34359739030,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7844896,"kb_used_data":5399444,"kb_used_omap":4927,"kb_used_meta":2440512,"kb_avail":86526944,"statfs":{"total":96636764160,"available":88603590656,"internally_reserved":0,"allocated":5529030656,"data_stored":65717789528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":2499084749},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,5,12,5,4,17,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":7.2190000000000003,"5min":4.1029999999999998,"15min":3.6440000000000001},"min":{"1min":0.35499999999999998,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":53.280000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":5.0759999999999996,"5min":3.3769999999999998,"15min":3.0670000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":52.982999999999997,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.35699999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:29:40 2023","interfaces":[{"interface":"back","average":{"1min":2.3170000000000002,"5min":2.8250000000000002,"15min":2.3540000000000001},"min":{"1min":0.219,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.436999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.97499999999999998},{"interface":"front","average":{"1min":2.3290000000000002,"5min":2.7120000000000002,"15min":2.8140000000000001},"min":{"1min":0.23499999999999999,"5min":0.215,"15min":0.186},"max":{"1min":12.286,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.25}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3406106624,"data_stored":44246834528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5524058112,"data_stored":65715606801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2121678848,"data_stored":21476169649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:30:38.168 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:38.168 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:38.168 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:38.168 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:38.169 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:30:40.333 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.324+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:40.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.338+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:40.469 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.464+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:40.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.464+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:40.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.464+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:40.502 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.497+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:40.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:40.512+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:43.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.153+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:43.172 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.169+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:43.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.234+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:43.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.234+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:43.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.248+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:43.249 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.248+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:43.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.314+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:43.315 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.314+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:43.328 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.327+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:43.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.330+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:43.353 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:43.334+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:44.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:44.929+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:44.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:44.987+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:45.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.000+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:45.057 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.056+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:45.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.737+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:45.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.737+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:45.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.798+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:45.799 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.798+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:45.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.843+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:45.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.843+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:45.844 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.843+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:45.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.883+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:45.945 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:45.941+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:47.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.842+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:47.843 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.842+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:47.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.869+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:47.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.869+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:47.871 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.869+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:47.926 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.925+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:47.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.948+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:47.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:47.948+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:48.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.115+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:48.118 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.115+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:48.124 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.123+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:48.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.151+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:48.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.190+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:48.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.210+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:48.213 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.210+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:48.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.218+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:48.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.218+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:48.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.218+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:48.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.262+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:30:48.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.262+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:48.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:48.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.262+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:48.266 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.262+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:48.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.324+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:48.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.354+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:48.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.354+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:48.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.377+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:48.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.377+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:48.435 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:48.430+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:49.405 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.404+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:49.481 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.473+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:49.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.481+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:49.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.485+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:49.507 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.506+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:49.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.530+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:49.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.530+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:49.537 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.536+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:49.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.602+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:49.607 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.602+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:49.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.724+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:49.731 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.724+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:49.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.738+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:49.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.738+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:49.741 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:49.738+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:51.185 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.183+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:51.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.189+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:51.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.196+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:51.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.196+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:51.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.196+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:51.218 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.217+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:51.305 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.304+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:51.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.493+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:30:51.495 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.493+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:51.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.513+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:51.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.513+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:51.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.596+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:51.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.596+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:51.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.596+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:51.602 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.601+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:51.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.631+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:51.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.666+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:51.738 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.735+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:51.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.766+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:51.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.766+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:51.767 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.766+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:51.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.791+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:51.960 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:51.960+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:52.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.694+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:30:52.695 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.694+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:30:52.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.769+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:30:52.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.769+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:52.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.825+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:52.827 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.825+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:52.828 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.825+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:52.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.885+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:52.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:52.926+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:53.003 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:30:53.000+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:30:53.009 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:30:53.003+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:30:53.453 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.444+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:53.453 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.444+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:53.453 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.444+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:53.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.444+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:53.501 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.500+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:53.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:53.520+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:54.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.398+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:30:54.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.411+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:30:54.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.411+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:30:54.466 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.465+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:30:54.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.493+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:54.495 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:54.494+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:58.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:30:58.997 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:30:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:30:59.033 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2169,"stamp":"2023-10-03T18:30:57.705579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24055412529,"num_objects":71221,"num_object_clones":65955,"num_object_copies":142442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71221,"num_whiteouts":0,"num_read":1510595,"num_read_kb":5456504,"num_write":1361822,"num_write_kb":5428231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80519,"ondisk_log_size":80519,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16494272,"kb_used_data":10859368,"kb_used_omap":9077,"kb_used_meta":5625802,"kb_avail":266621248,"statfs":{"total":289910292480,"available":273020157952,"internally_reserved":0,"allocated":11119992832,"data_stored":131337833623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9295696,"internal_metadata":5760821424},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,2,5,5,45,10,4,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14684160,"num_objects":-78,"num_object_clones":-78,"num_object_copies":-156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-78,"num_whiteouts":0,"num_read":3733,"num_read_kb":14442,"num_write":3553,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010852"},"pg_stats":[{"pgid":"2.7","version":"431'207665","reported_seq":362767,"reported_epoch":431,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:55.032167+0000","last_change":"2023-10-03T18:30:53.072019+0000","last_active":"2023-10-03T18:30:55.032167+0000","last_peered":"2023-10-03T18:30:55.032167+0000","last_clean":"2023-10-03T18:30:55.032167+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:30:55.032167+0000","last_undegraded":"2023-10-03T18:30:55.032167+0000","last_fullsized":"2023-10-03T18:30:55.032167+0000","mapping_epoch":12,"log_start":"386'197600","ondisk_log_start":"386'197600","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":322,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:37:33.952265+0000","scrub_duration":505.38444223300002,"objects_trimmed":47,"snaptrim_duration":2.2339288059999998,"stat_sum":{"num_bytes":2996502963,"num_objects":12953,"num_object_clones":12303,"num_object_copies":25906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12953,"num_whiteouts":0,"num_read":183946,"num_read_kb":678589,"num_write":169165,"num_write_kb":674571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ca","length":"1"}]},{"pgid":"2.6","version":"431'192141","reported_seq":336485,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:55.043663+0000","last_change":"2023-10-03T18:30:51.907315+0000","last_active":"2023-10-03T18:30:55.043663+0000","last_peered":"2023-10-03T18:30:55.043663+0000","last_clean":"2023-10-03T18:30:55.043663+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:30:55.043663+0000","last_undegraded":"2023-10-03T18:30:55.043663+0000","last_fullsized":"2023-10-03T18:30:55.043663+0000","mapping_epoch":12,"log_start":"383'182075","ondisk_log_start":"383'182075","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":235,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":27,"snaptrim_duration":4.1932712060000004,"stat_sum":{"num_bytes":2768364243,"num_objects":9526,"num_object_clones":8920,"num_object_copies":19052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9526,"num_whiteouts":0,"num_read":170586,"num_read_kb":626205,"num_write":156235,"num_write_kb":623536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"5"}]},{"pgid":"2.5","version":"431'230478","reported_seq":400733,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:55.042325+0000","last_change":"2023-10-03T18:30:53.066765+0000","last_active":"2023-10-03T18:30:55.042325+0000","last_peered":"2023-10-03T18:30:55.042325+0000","last_clean":"2023-10-03T18:30:55.042325+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:30:55.042325+0000","last_undegraded":"2023-10-03T18:30:55.042325+0000","last_fullsized":"2023-10-03T18:30:55.042325+0000","mapping_epoch":12,"log_start":"395'220477","ondisk_log_start":"395'220477","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":270,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":5.3528023620000003,"stat_sum":{"num_bytes":3247751721,"num_objects":12333,"num_object_clones":11625,"num_object_copies":24666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12333,"num_whiteouts":0,"num_read":210963,"num_read_kb":741972,"num_write":184350,"num_write_kb":733579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"}]},{"pgid":"2.0","version":"431'228143","reported_seq":387932,"reported_epoch":431,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:55.954601+0000","last_change":"2023-10-03T18:30:54.623663+0000","last_active":"2023-10-03T18:30:55.954601+0000","last_peered":"2023-10-03T18:30:55.954601+0000","last_clean":"2023-10-03T18:30:55.954601+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:30:55.954601+0000","last_undegraded":"2023-10-03T18:30:55.954601+0000","last_fullsized":"2023-10-03T18:30:55.954601+0000","mapping_epoch":12,"log_start":"400'218096","ondisk_log_start":"400'218096","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":162,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:40:37.900562+0000","scrub_duration":734.24538164700004,"objects_trimmed":132,"snaptrim_duration":6.900786138,"stat_sum":{"num_bytes":3137585909,"num_objects":8417,"num_object_clones":7730,"num_object_copies":16834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8417,"num_whiteouts":0,"num_read":191325,"num_read_kb":707206,"num_write":177833,"num_write_kb":709229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"6"}]},{"pgid":"1.0","version":"88'99","reported_seq":597,"reported_epoch":430,"state":"active+clean","last_fresh":"2023-10-03T18:30:42.715421+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:30:42.715421+0000","last_peered":"2023-10-03T18:30:42.715421+0000","last_clean":"2023-10-03T18:30:42.715421+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:30:42.715421+0000","last_undegraded":"2023-10-03T18:30:42.715421+0000","last_fullsized":"2023-10-03T18:30:42.715421+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"431'221716","reported_seq":384909,"reported_epoch":431,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:30:54.993327+0000","last_change":"2023-10-03T18:30:54.363440+0000","last_active":"2023-10-03T18:30:54.993327+0000","last_peered":"2023-10-03T18:30:54.993327+0000","last_clean":"2023-10-03T18:30:54.993327+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:30:54.993327+0000","last_undegraded":"2023-10-03T18:30:54.993327+0000","last_fullsized":"2023-10-03T18:30:54.993327+0000","mapping_epoch":12,"log_start":"390'211695","ondisk_log_start":"390'211695","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":260,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":35,"snaptrim_duration":6.5678831569999998,"stat_sum":{"num_bytes":3190276786,"num_objects":11797,"num_object_clones":11101,"num_object_copies":23594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11797,"num_whiteouts":0,"num_read":194757,"num_read_kb":719325,"num_write":180112,"num_write_kb":718266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"}]},{"pgid":"2.1","version":"431'219285","reported_seq":367191,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-10-03T18:30:55.955705+0000","last_change":"2023-10-03T18:30:42.746558+0000","last_active":"2023-10-03T18:30:55.955705+0000","last_peered":"2023-10-03T18:30:55.955705+0000","last_clean":"2023-10-03T18:30:55.955705+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:30:55.955705+0000","last_undegraded":"2023-10-03T18:30:55.955705+0000","last_fullsized":"2023-10-03T18:30:55.955705+0000","mapping_epoch":12,"log_start":"396'209201","ondisk_log_start":"396'209201","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:19:19.195819+0000","scrub_duration":424.08306342600002,"objects_trimmed":0,"snaptrim_duration":0.046177874000000001,"stat_sum":{"num_bytes":2907496970,"num_objects":2994,"num_object_clones":2361,"num_object_copies":5988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":184097,"num_read_kb":658068,"num_write":164599,"num_write_kb":655551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"}]},{"pgid":"2.2","version":"431'219415","reported_seq":373149,"reported_epoch":431,"state":"active+clean","last_fresh":"2023-10-03T18:30:57.561657+0000","last_change":"2023-10-03T18:30:43.236564+0000","last_active":"2023-10-03T18:30:57.561657+0000","last_peered":"2023-10-03T18:30:57.561657+0000","last_clean":"2023-10-03T18:30:57.561657+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:30:57.561657+0000","last_undegraded":"2023-10-03T18:30:57.561657+0000","last_fullsized":"2023-10-03T18:30:57.561657+0000","mapping_epoch":12,"log_start":"396'209349","ondisk_log_start":"396'209349","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10066,"log_dups_size":0,"ondisk_log_size":10066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:06:25.748643+0000","scrub_duration":370.574046178,"objects_trimmed":7,"snaptrim_duration":0.515026282,"stat_sum":{"num_bytes":2910446492,"num_objects":2995,"num_object_clones":2350,"num_object_copies":5990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":194974,"num_read_kb":671165,"num_write":165884,"num_write_kb":659711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"}]},{"pgid":"2.4","version":"431'202754","reported_seq":353311,"reported_epoch":431,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:30:55.021997+0000","last_change":"2023-10-03T18:30:54.385367+0000","last_active":"2023-10-03T18:30:55.021997+0000","last_peered":"2023-10-03T18:30:55.021997+0000","last_clean":"2023-10-03T18:30:55.021997+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:30:55.021997+0000","last_undegraded":"2023-10-03T18:30:55.021997+0000","last_fullsized":"2023-10-03T18:30:55.021997+0000","mapping_epoch":12,"log_start":"385'192684","ondisk_log_start":"385'192684","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":243,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":44,"snaptrim_duration":3.0832540239999999,"stat_sum":{"num_bytes":2896397077,"num_objects":10204,"num_object_clones":9565,"num_object_copies":20408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10204,"num_whiteouts":0,"num_read":179841,"num_read_kb":653883,"num_write":163507,"num_write_kb":651894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ca","length":"1"},{"start":"1cc","length":"2"},{"start":"1d6","length":"a"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24054822161,"num_objects":71219,"num_object_clones":65955,"num_object_copies":142438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71219,"num_whiteouts":0,"num_read":1510489,"num_read_kb":5456413,"num_write":1361685,"num_write_kb":5426337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11105804288,"data_stored":131311516194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74750,"internal_metadata":0},"log_size":80420,"ondisk_log_size":80420,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739066,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3869788,"kb_used_data":2086356,"kb_used_omap":1375,"kb_used_meta":1782048,"kb_avail":90502052,"statfs":{"total":96636764160,"available":92674101248,"internally_reserved":0,"allocated":2136428544,"data_stored":21345965848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1824817284},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,0,2,4,3,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.0840000000000001,"15min":2.298},"min":{"1min":0.17199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.401999999999999,"5min":14.26,"15min":15.964},"last":0.24199999999999999},{"interface":"front","average":{"1min":1.431,"5min":2.5230000000000001,"15min":2.3559999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.6699999999999999,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.36799999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":2.6899999999999999,"15min":2.4089999999999998},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":9.3219999999999992,"5min":15.85,"15min":17.274000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":2.2080000000000002,"5min":2.3420000000000001,"15min":2.2559999999999998},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.1479999999999997,"5min":13.842000000000001,"15min":13.842000000000001},"last":0.38700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739065,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4755840,"kb_used_data":3347512,"kb_used_omap":2774,"kb_used_meta":1405545,"kb_avail":89616000,"statfs":{"total":96636764160,"available":91766784000,"internally_reserved":0,"allocated":3427852288,"data_stored":44300795623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1439278687},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.847,"5min":3.2639999999999998,"15min":3.016},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":7.8369999999999997,"5min":21.256,"15min":45.886000000000003},"last":4.327},{"interface":"front","average":{"1min":4.9530000000000003,"5min":4.0350000000000001,"15min":3.4220000000000002},"min":{"1min":0.34799999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":20.763000000000002,"5min":20.763000000000002,"15min":45.555},"last":1.0029999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.0289999999999999,"15min":1.966},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":4.9269999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.30099999999999999},{"interface":"front","average":{"1min":3.0499999999999998,"5min":3.4980000000000002,"15min":2.6099999999999999},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":18.352,"5min":25.77,"15min":25.77},"last":7.1130000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739034,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7868644,"kb_used_data":5425500,"kb_used_omap":4927,"kb_used_meta":2438208,"kb_avail":86503196,"statfs":{"total":96636764160,"available":88579272704,"internally_reserved":0,"allocated":5555712000,"data_stored":65691072152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5045811,"internal_metadata":2496725453},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,5,1,38,5,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":4.9630000000000001,"5min":4.4580000000000002,"15min":3.8260000000000001},"min":{"1min":0.27800000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":23.126000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":2.7280000000000002},{"interface":"front","average":{"1min":4.1379999999999999,"5min":3.5979999999999999,"15min":3.1659999999999999},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":22.905000000000001,"5min":52.982999999999997,"15min":52.982999999999997},"last":10.664}]},{"osd":2,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":2.641,"5min":2.956,"15min":2.3879999999999999},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":12.904999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":3.5880000000000001},{"interface":"front","average":{"1min":2.1030000000000002,"5min":2.6970000000000001,"15min":2.754},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.3490000000000002,"5min":36.332999999999998,"15min":36.332999999999998},"last":14.926}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3422560256,"data_stored":44298108256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5551472640,"data_stored":65667773713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32203,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2131771392,"data_stored":21345634225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:30:59.035 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:59.035 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:59.035 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:59.035 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:30:59.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:30:59.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:59.603+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:30:59.748 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:30:59.747+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:30:59.799 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:30:59.818 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:31:00.569 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:31:00.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:31:01.349 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:31:01.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:31:02.037 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:02.030+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:31:02.048 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:02.040+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:31:02.049 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:02.046+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:31:02.079 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.079+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:02.090 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.090+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:02.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.095+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:02.144 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:31:02.163 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:31:02.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.177+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:02.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.177+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:02.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.177+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:02.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.177+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:02.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.177+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:02.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.186+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:02.187 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.186+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:02.314 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.314+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:02.347 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.346+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:02.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.571+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:02.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.571+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:02.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.571+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:02.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.571+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:02.573 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.571+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:02.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.601+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:02.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.601+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:02.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.601+0000 7fc515f98700 -1 reset not still connected to 0x56221940cea0 2023-10-03T18:31:02.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.601+0000 7fc515f98700 -1 reset not still connected to 0x56221b491ba0 2023-10-03T18:31:02.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.601+0000 7fc515f98700 -1 reset not still connected to 0x5622204a01a0 2023-10-03T18:31:02.614 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.612+0000 7fc515f98700 -1 reset not still connected to 0x562226215380 2023-10-03T18:31:02.787 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.785+0000 7fc515f98700 -1 reset not still connected to 0x56222a181d40 2023-10-03T18:31:02.873 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:02.871+0000 7fc515f98700 -1 reset not still connected to 0x562233088dd0 2023-10-03T18:31:04.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:04.029+0000 7fc515f98700 -1 reset not still connected to 0x56222a181d40 2023-10-03T18:31:07.641 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.639+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:07.702 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:31:07.701+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:31:07.702 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:31:07.701+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:31:07.702 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:31:07.701+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:31:07.703 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:31:07.701+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:31:07.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.865+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:07.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.879+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:07.884 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.879+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:07.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.879+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:07.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.940+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:07.941 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:07.940+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:08.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:08.008+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:08.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:08.024+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:08.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:08.030+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:11.361 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.359+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:11.389 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.388+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:11.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.390+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:11.568 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:11.567+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:31:11.568 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:11.567+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:31:11.568 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:11.567+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:31:11.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.650+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:11.669 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.668+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:11.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.668+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:11.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.701+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:11.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.701+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:11.714 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.713+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:11.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.763+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:11.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:11.763+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:12.930 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:12.929+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:12.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:12.940+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:12.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:12.954+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:13.047 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.047+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:13.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.060+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:13.074 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.074+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:13.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.194+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:13.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.194+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:13.227 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.225+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:13.323 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.320+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:13.323 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.320+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:13.565 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.564+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:13.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.584+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:13.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.601+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:13.609 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.601+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:13.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.601+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:13.652 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.651+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:13.720 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:13.719+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:17.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.651+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:17.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.768+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:17.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.769+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:17.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.769+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:17.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.769+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:17.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.769+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:17.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.769+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:17.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.770+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:17.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:17.799+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:20.624 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.623+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:20.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.676+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:20.679 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.676+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:20.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.738+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:20.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.738+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:20.847 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.846+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:20.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.846+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:20.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.846+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:20.875 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.872+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:20.900 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:20.894+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:21.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.616+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:21.618 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.616+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:21.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.693+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:21.837 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.836+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:21.838 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.836+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:21.869 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.865+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:21.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.929+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:21.985 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:21.981+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:22.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:31:22.850 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:31:22.870 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:31:22.888 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:22.878+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:22.889 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2190,"stamp":"2023-10-03T18:31:22.707458+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24082171697,"num_objects":72707,"num_object_clones":67441,"num_object_copies":145414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72707,"num_whiteouts":0,"num_read":1521699,"num_read_kb":5486933,"num_write":1368533,"num_write_kb":5454355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80480,"ondisk_log_size":80480,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15490576,"kb_used_data":10909816,"kb_used_omap":8643,"kb_used_meta":4572092,"kb_avail":267624944,"statfs":{"total":289910292480,"available":274047942656,"internally_reserved":0,"allocated":11171651584,"data_stored":134671587905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8851301,"internal_metadata":4681822363},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,6,12,6,14,9,16],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9736192,"num_objects":811,"num_object_clones":811,"num_object_copies":1622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":811,"num_whiteouts":0,"num_read":2526,"num_read_kb":10036,"num_write":2373,"num_write_kb":9492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001374"},"pg_stats":[{"pgid":"2.7","version":"440'209099","reported_seq":364765,"reported_epoch":440,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:22.572758+0000","last_change":"2023-10-03T18:31:16.588866+0000","last_active":"2023-10-03T18:31:22.572758+0000","last_peered":"2023-10-03T18:31:22.572758+0000","last_clean":"2023-10-03T18:31:22.572758+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:31:22.572758+0000","last_undegraded":"2023-10-03T18:31:22.572758+0000","last_fullsized":"2023-10-03T18:31:22.572758+0000","mapping_epoch":12,"log_start":"396'199004","ondisk_log_start":"396'199004","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10095,"log_dups_size":0,"ondisk_log_size":10095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":338,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:52:43.872351+0000","scrub_duration":505.38444223300002,"objects_trimmed":29,"snaptrim_duration":3.8538963539999997,"stat_sum":{"num_bytes":2999943603,"num_objects":13219,"num_object_clones":12569,"num_object_copies":26438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13219,"num_whiteouts":0,"num_read":185208,"num_read_kb":682292,"num_write":170025,"num_write_kb":677946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d7","length":"1"}]},{"pgid":"2.6","version":"440'193498","reported_seq":338479,"reported_epoch":440,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:22.531394+0000","last_change":"2023-10-03T18:31:20.355887+0000","last_active":"2023-10-03T18:31:22.531394+0000","last_peered":"2023-10-03T18:31:22.531394+0000","last_clean":"2023-10-03T18:31:22.531394+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:31:22.531394+0000","last_undegraded":"2023-10-03T18:31:22.531394+0000","last_fullsized":"2023-10-03T18:31:22.531394+0000","mapping_epoch":12,"log_start":"393'183482","ondisk_log_start":"393'183482","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":251,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":7.6307395319999998,"stat_sum":{"num_bytes":2771518061,"num_objects":9760,"num_object_clones":9156,"num_object_copies":19520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9760,"num_whiteouts":0,"num_read":171789,"num_read_kb":629837,"num_write":157015,"num_write_kb":626605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"9"}]},{"pgid":"2.5","version":"440'232103","reported_seq":403114,"reported_epoch":440,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:22.505997+0000","last_change":"2023-10-03T18:31:20.406246+0000","last_active":"2023-10-03T18:31:22.505997+0000","last_peered":"2023-10-03T18:31:22.505997+0000","last_clean":"2023-10-03T18:31:22.505997+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:31:22.505997+0000","last_undegraded":"2023-10-03T18:31:22.505997+0000","last_fullsized":"2023-10-03T18:31:22.505997+0000","mapping_epoch":12,"log_start":"398'222085","ondisk_log_start":"398'222085","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10018,"log_dups_size":0,"ondisk_log_size":10018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":287,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":53,"snaptrim_duration":3.195275224,"stat_sum":{"num_bytes":3251610255,"num_objects":12610,"num_object_clones":11900,"num_object_copies":25220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12610,"num_whiteouts":0,"num_read":212766,"num_read_kb":746510,"num_write":185325,"num_write_kb":737343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"3"}]},{"pgid":"2.0","version":"440'230030","reported_seq":390334,"reported_epoch":440,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:21.123054+0000","last_change":"2023-10-03T18:31:18.975679+0000","last_active":"2023-10-03T18:31:21.123054+0000","last_peered":"2023-10-03T18:31:21.123054+0000","last_clean":"2023-10-03T18:31:21.123054+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:31:21.123054+0000","last_undegraded":"2023-10-03T18:31:21.123054+0000","last_fullsized":"2023-10-03T18:31:21.123054+0000","mapping_epoch":12,"log_start":"410'220007","ondisk_log_start":"410'220007","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":171,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:22.954032+0000","scrub_duration":734.24538164700004,"objects_trimmed":117,"snaptrim_duration":6.2429718029999997,"stat_sum":{"num_bytes":3141010115,"num_objects":8437,"num_object_clones":7751,"num_object_copies":16874,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8437,"num_whiteouts":0,"num_read":192612,"num_read_kb":710999,"num_write":178691,"num_write_kb":712585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":606,"reported_epoch":439,"state":"active+clean","last_fresh":"2023-10-03T18:31:07.927934+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:31:07.927934+0000","last_peered":"2023-10-03T18:31:07.927934+0000","last_clean":"2023-10-03T18:31:07.927934+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:31:07.927934+0000","last_undegraded":"2023-10-03T18:31:07.927934+0000","last_fullsized":"2023-10-03T18:31:07.927934+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"440'223289","reported_seq":387149,"reported_epoch":440,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:22.513829+0000","last_change":"2023-10-03T18:31:19.591862+0000","last_active":"2023-10-03T18:31:22.513829+0000","last_peered":"2023-10-03T18:31:22.513829+0000","last_clean":"2023-10-03T18:31:22.513829+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:31:22.513829+0000","last_undegraded":"2023-10-03T18:31:22.513829+0000","last_fullsized":"2023-10-03T18:31:22.513829+0000","mapping_epoch":12,"log_start":"397'213200","ondisk_log_start":"397'213200","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":277,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":39,"snaptrim_duration":1.5973131280000001,"stat_sum":{"num_bytes":3194151653,"num_objects":12063,"num_object_clones":11366,"num_object_copies":24126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12063,"num_whiteouts":0,"num_read":196142,"num_read_kb":723503,"num_write":181074,"num_write_kb":722053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"3"}]},{"pgid":"2.1","version":"440'220908","reported_seq":369360,"reported_epoch":440,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:21.091260+0000","last_change":"2023-10-03T18:31:19.367137+0000","last_active":"2023-10-03T18:31:21.091260+0000","last_peered":"2023-10-03T18:31:21.091260+0000","last_clean":"2023-10-03T18:31:21.091260+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:31:21.091260+0000","last_undegraded":"2023-10-03T18:31:21.091260+0000","last_fullsized":"2023-10-03T18:31:21.091260+0000","mapping_epoch":12,"log_start":"399'210810","ondisk_log_start":"399'210810","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:50:34.507248+0000","scrub_duration":424.08306342600002,"objects_trimmed":114,"snaptrim_duration":6.6274621929999995,"stat_sum":{"num_bytes":2910691850,"num_objects":3093,"num_object_clones":2460,"num_object_copies":6186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3093,"num_whiteouts":0,"num_read":185527,"num_read_kb":661772,"num_write":165397,"num_write_kb":658649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"7"}]},{"pgid":"2.2","version":"440'220695","reported_seq":374949,"reported_epoch":440,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:17.622736+0000","last_change":"2023-10-03T18:31:15.221980+0000","last_active":"2023-10-03T18:31:17.622736+0000","last_peered":"2023-10-03T18:31:17.622736+0000","last_clean":"2023-10-03T18:31:17.622736+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:31:17.622736+0000","last_undegraded":"2023-10-03T18:31:17.622736+0000","last_fullsized":"2023-10-03T18:31:17.622736+0000","mapping_epoch":12,"log_start":"398'210656","ondisk_log_start":"398'210656","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:21.025422+0000","scrub_duration":370.574046178,"objects_trimmed":65,"snaptrim_duration":2.5096452149999999,"stat_sum":{"num_bytes":2912797545,"num_objects":3073,"num_object_clones":2429,"num_object_copies":6146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3073,"num_whiteouts":0,"num_read":196374,"num_read_kb":674283,"num_write":166504,"num_write_kb":662043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"6"}]},{"pgid":"2.4","version":"440'204193","reported_seq":355401,"reported_epoch":440,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:22.505754+0000","last_change":"2023-10-03T18:31:19.493102+0000","last_active":"2023-10-03T18:31:22.505754+0000","last_peered":"2023-10-03T18:31:22.505754+0000","last_clean":"2023-10-03T18:31:22.505754+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:31:22.505754+0000","last_undegraded":"2023-10-03T18:31:22.505754+0000","last_fullsized":"2023-10-03T18:31:22.505754+0000","mapping_epoch":12,"log_start":"396'194190","ondisk_log_start":"396'194190","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":259,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":29,"snaptrim_duration":6.7680602050000003,"stat_sum":{"num_bytes":2899858247,"num_objects":10450,"num_object_clones":9810,"num_object_copies":20900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10450,"num_whiteouts":0,"num_read":181175,"num_read_kb":657646,"num_write":164365,"num_write_kb":655237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d7","length":"9"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24081581329,"num_objects":72705,"num_object_clones":67441,"num_object_copies":145410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72705,"num_whiteouts":0,"num_read":1521593,"num_read_kb":5486842,"num_write":1368396,"num_write_kb":5452461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11156058112,"data_stored":134703884834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78030,"internal_metadata":0},"log_size":80381,"ondisk_log_size":80381,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739071,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3929592,"kb_used_data":2096240,"kb_used_omap":1375,"kb_used_meta":1831968,"kb_avail":90442248,"statfs":{"total":96636764160,"available":92612861952,"internally_reserved":0,"allocated":2146549760,"data_stored":21810613159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1875935364},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,4,2,0,2,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1890000000000001,"5min":2.0840000000000001,"15min":2.298},"min":{"1min":0.17199999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":10.401999999999999,"5min":14.26,"15min":15.964},"last":1.958},{"interface":"front","average":{"1min":1.431,"5min":2.5230000000000001,"15min":2.3559999999999999},"min":{"1min":0.218,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.6699999999999999,"5min":34.176000000000002,"15min":34.176000000000002},"last":0.19500000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:30:28 2023","interfaces":[{"interface":"back","average":{"1min":2.5779999999999998,"5min":2.6899999999999999,"15min":2.4089999999999998},"min":{"1min":0.27600000000000002,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":9.3219999999999992,"5min":15.85,"15min":17.274000000000001},"last":3.1040000000000001},{"interface":"front","average":{"1min":2.2080000000000002,"5min":2.3420000000000001,"15min":2.2559999999999998},"min":{"1min":0.24199999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":9.1479999999999997,"5min":13.842000000000001,"15min":13.842000000000001},"last":2.2879999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739069,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4853800,"kb_used_data":3359392,"kb_used_omap":2774,"kb_used_meta":1491625,"kb_avail":89518040,"statfs":{"total":96636764160,"available":91666472960,"internally_reserved":0,"allocated":3440017408,"data_stored":45377303412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1527424607},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,4,0,5,3,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.847,"5min":3.2639999999999998,"15min":3.016},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":7.8369999999999997,"5min":21.256,"15min":45.886000000000003},"last":1.0229999999999999},{"interface":"front","average":{"1min":4.9530000000000003,"5min":4.0350000000000001,"15min":3.4220000000000002},"min":{"1min":0.34799999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":20.763000000000002,"5min":20.763000000000002,"15min":45.555},"last":0.433}]},{"osd":2,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.0289999999999999,"15min":1.966},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":4.9269999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.79600000000000004},{"interface":"front","average":{"1min":3.0499999999999998,"5min":3.4980000000000002,"15min":2.6099999999999999},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":18.352,"5min":25.77,"15min":25.77},"last":0.28999999999999998}]}]},{"osd":1,"up_from":8,"seq":34359739039,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6707184,"kb_used_data":5454184,"kb_used_omap":4493,"kb_used_meta":1248498,"kb_avail":87664656,"statfs":{"total":96636764160,"available":89768607744,"internally_reserved":0,"allocated":5585084416,"data_stored":67483671334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1278462392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,6,4,4,9,4,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":4.9630000000000001,"5min":4.4580000000000002,"15min":3.8260000000000001},"min":{"1min":0.27800000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":23.126000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":2.5369999999999999},{"interface":"front","average":{"1min":4.1379999999999999,"5min":3.5979999999999999,"15min":3.1659999999999999},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":22.905000000000001,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.49299999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":2.641,"5min":2.956,"15min":2.3879999999999999},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":12.904999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":2.1030000000000002,"5min":2.6970000000000001,"15min":2.754},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.3490000000000002,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.53100000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3434479616,"data_stored":45377719647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5579796480,"data_stored":67502908689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2141782016,"data_stored":21823256498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:31:22.890 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:22.891 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:22.891 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:22.891 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:22.891 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:31:22.952 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:22.947+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:23.023 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.021+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:23.024 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.021+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:23.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.046+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:23.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.046+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:23.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.158+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:23.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.862+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:24.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.983+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:24.041 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:23.983+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:24.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.031+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:24.046 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.041+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:24.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.041+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:24.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.041+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:24.121 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.041+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:24.121 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.041+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:24.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:24.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:24.917+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:25.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:25.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:25.160+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:29.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.893+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:29.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.935+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:29.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.942+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:29.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.942+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:29.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.955+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:29.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.955+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:29.963 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.955+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:29.968 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:29.966+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:30.087 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:30.084+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:31.108 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.106+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:31.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.164+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:31.214 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.212+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:31.214 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.212+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:31.258 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.257+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:31.258 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.257+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:31.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.376+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:31.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.376+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:31.378 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:31.376+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:32.468 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.462+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:32.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.523+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:32.525 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.523+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:32.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.581+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:32.583 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:32.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:32.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:32.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:32.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:32.584 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:32.582+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:33.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:33.937+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:33.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:33.937+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:33.939 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:33.937+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:33.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:33.994+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:33.997 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:33.994+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:34.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.040+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:34.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.040+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:34.042 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.040+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:34.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.063+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:34.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.082+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:34.137 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.134+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:34.232 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.230+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:34.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.326+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:34.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.326+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:34.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.326+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:34.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.326+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:34.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.326+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:34.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.327+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:34.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:34.327+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:36.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.723+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:36.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.729+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:36.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.746+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:36.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.746+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:36.812 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.809+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:36.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.852+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:36.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:36.863+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:40.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:40.873+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:40.911 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:40.909+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:40.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:40.943+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:42.233 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:42.231+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:31:42.321 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.317+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:42.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.406+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:42.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.473+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:42.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.473+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:42.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.473+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:42.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.527+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:42.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:42.546+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:42.893 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:31:43.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.127+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:43.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.128+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:43.130 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.128+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:43.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.170+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:43.197 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.192+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:43.243 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.241+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:43.276 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.274+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:43.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.274+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:43.277 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.275+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:43.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.339+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:43.373 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:43.370+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:43.623 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:31:43.647 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:31:43.656 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2203,"stamp":"2023-10-03T18:31:43.499108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24102217521,"num_objects":72844,"num_object_clones":67578,"num_object_copies":145688,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72844,"num_whiteouts":0,"num_read":1526694,"num_read_kb":5506789,"num_write":1373307,"num_write_kb":5473451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80527,"ondisk_log_size":80527,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15655420,"kb_used_data":10950048,"kb_used_omap":8643,"kb_used_meta":4696700,"kb_avail":267460100,"statfs":{"total":289910292480,"available":273879142400,"internally_reserved":0,"allocated":11212849152,"data_stored":135639139665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8851301,"internal_metadata":4809420955},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,1,8,11,12,12,14],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12152832,"num_objects":-18,"num_object_clones":-18,"num_object_copies":-36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-18,"num_whiteouts":0,"num_read":3013,"num_read_kb":11984,"num_write":2848,"num_write_kb":11392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004501"},"pg_stats":[{"pgid":"2.7","version":"443'209996","reported_seq":366031,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:39.350761+0000","last_change":"2023-10-03T18:31:37.837727+0000","last_active":"2023-10-03T18:31:39.350761+0000","last_peered":"2023-10-03T18:31:39.350761+0000","last_clean":"2023-10-03T18:31:39.350761+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:31:39.350761+0000","last_undegraded":"2023-10-03T18:31:39.350761+0000","last_fullsized":"2023-10-03T18:31:39.350761+0000","mapping_epoch":12,"log_start":"398'199907","ondisk_log_start":"398'199907","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10089,"log_dups_size":0,"ondisk_log_size":10089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":336,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:52:43.872351+0000","scrub_duration":505.38444223300002,"objects_trimmed":24,"snaptrim_duration":0.51081489199999996,"stat_sum":{"num_bytes":3002139059,"num_objects":13302,"num_object_clones":12652,"num_object_copies":26604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13302,"num_whiteouts":0,"num_read":185767,"num_read_kb":684500,"num_write":170557,"num_write_kb":680074,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"443'194270","reported_seq":339596,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:39.241485+0000","last_change":"2023-10-03T18:31:39.239713+0000","last_active":"2023-10-03T18:31:39.241485+0000","last_peered":"2023-10-03T18:31:39.241485+0000","last_clean":"2023-10-03T18:31:39.241485+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:31:39.241485+0000","last_undegraded":"2023-10-03T18:31:39.241485+0000","last_fullsized":"2023-10-03T18:31:39.241485+0000","mapping_epoch":12,"log_start":"396'184183","ondisk_log_start":"396'184183","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":249,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.5496473339999999,"stat_sum":{"num_bytes":2773635693,"num_objects":9875,"num_object_clones":9271,"num_object_copies":19750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9875,"num_whiteouts":0,"num_read":172296,"num_read_kb":631849,"num_write":157531,"num_write_kb":628669,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"1"}]},{"pgid":"2.5","version":"443'233059","reported_seq":404534,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:39.278400+0000","last_change":"2023-10-03T18:31:37.380492+0000","last_active":"2023-10-03T18:31:39.278400+0000","last_peered":"2023-10-03T18:31:39.278400+0000","last_clean":"2023-10-03T18:31:39.278400+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:31:39.278400+0000","last_undegraded":"2023-10-03T18:31:39.278400+0000","last_fullsized":"2023-10-03T18:31:39.278400+0000","mapping_epoch":12,"log_start":"401'222986","ondisk_log_start":"401'222986","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10073,"log_dups_size":0,"ondisk_log_size":10073,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":284,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.5898931919999999,"stat_sum":{"num_bytes":3254084239,"num_objects":12726,"num_object_clones":12016,"num_object_copies":25452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12726,"num_whiteouts":0,"num_read":213422,"num_read_kb":749126,"num_write":185910,"num_write_kb":739683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"6"}]},{"pgid":"2.0","version":"443'231470","reported_seq":392192,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:41.224019+0000","last_change":"2023-10-03T18:31:40.374994+0000","last_active":"2023-10-03T18:31:41.224019+0000","last_peered":"2023-10-03T18:31:41.224019+0000","last_clean":"2023-10-03T18:31:41.224019+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:31:41.224019+0000","last_undegraded":"2023-10-03T18:31:41.224019+0000","last_fullsized":"2023-10-03T18:31:41.224019+0000","mapping_epoch":12,"log_start":"413'221415","ondisk_log_start":"413'221415","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10055,"log_dups_size":0,"ondisk_log_size":10055,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":162,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:50:22.954032+0000","scrub_duration":734.24538164700004,"objects_trimmed":54,"snaptrim_duration":2.6303973049999998,"stat_sum":{"num_bytes":3143770819,"num_objects":8342,"num_object_clones":7656,"num_object_copies":16684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8342,"num_whiteouts":0,"num_read":193320,"num_read_kb":713827,"num_write":179340,"num_write_kb":715181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":609,"reported_epoch":442,"state":"active+clean","last_fresh":"2023-10-03T18:31:32.807584+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:31:32.807584+0000","last_peered":"2023-10-03T18:31:32.807584+0000","last_clean":"2023-10-03T18:31:32.807584+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:31:32.807584+0000","last_undegraded":"2023-10-03T18:31:32.807584+0000","last_fullsized":"2023-10-03T18:31:32.807584+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"443'224209","reported_seq":388496,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:39.270907+0000","last_change":"2023-10-03T18:31:37.660577+0000","last_active":"2023-10-03T18:31:39.270907+0000","last_peered":"2023-10-03T18:31:39.270907+0000","last_clean":"2023-10-03T18:31:39.270907+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:31:39.270907+0000","last_undegraded":"2023-10-03T18:31:39.270907+0000","last_fullsized":"2023-10-03T18:31:39.270907+0000","mapping_epoch":12,"log_start":"400'214204","ondisk_log_start":"400'214204","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10005,"log_dups_size":0,"ondisk_log_size":10005,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":274,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.8540241330000002,"stat_sum":{"num_bytes":3196560101,"num_objects":12156,"num_object_clones":11459,"num_object_copies":24312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12156,"num_whiteouts":0,"num_read":196757,"num_read_kb":725947,"num_write":181634,"num_write_kb":724293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"6"}]},{"pgid":"2.1","version":"443'222362","reported_seq":371152,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:41.253809+0000","last_change":"2023-10-03T18:31:41.046336+0000","last_active":"2023-10-03T18:31:41.253809+0000","last_peered":"2023-10-03T18:31:41.253809+0000","last_clean":"2023-10-03T18:31:41.253809+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:31:41.253809+0000","last_undegraded":"2023-10-03T18:31:41.253809+0000","last_fullsized":"2023-10-03T18:31:41.253809+0000","mapping_epoch":12,"log_start":"410'212328","ondisk_log_start":"410'212328","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10034,"log_dups_size":0,"ondisk_log_size":10034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:50:34.507248+0000","scrub_duration":424.08306342600002,"objects_trimmed":50,"snaptrim_duration":3.3019138730000002,"stat_sum":{"num_bytes":2913247754,"num_objects":2954,"num_object_clones":2321,"num_object_copies":5908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2954,"num_whiteouts":0,"num_read":186142,"num_read_kb":664212,"num_write":166025,"num_write_kb":661161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"}]},{"pgid":"2.2","version":"443'222447","reported_seq":377139,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:31:42.759164+0000","last_change":"2023-10-03T18:31:42.105836+0000","last_active":"2023-10-03T18:31:42.759164+0000","last_peered":"2023-10-03T18:31:42.759164+0000","last_clean":"2023-10-03T18:31:42.759164+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:31:42.759164+0000","last_undegraded":"2023-10-03T18:31:42.759164+0000","last_fullsized":"2023-10-03T18:31:42.759164+0000","mapping_epoch":12,"log_start":"410'212363","ondisk_log_start":"410'212363","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10084,"log_dups_size":0,"ondisk_log_size":10084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:21.025422+0000","scrub_duration":370.574046178,"objects_trimmed":75,"snaptrim_duration":4.3480113459999998,"stat_sum":{"num_bytes":2915988329,"num_objects":2973,"num_object_clones":2329,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":197169,"num_read_kb":677443,"num_write":167259,"num_write_kb":665063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"}]},{"pgid":"2.4","version":"443'205097","reported_seq":356671,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:31:39.299994+0000","last_change":"2023-10-03T18:31:39.237593+0000","last_active":"2023-10-03T18:31:39.299994+0000","last_peered":"2023-10-03T18:31:39.299994+0000","last_clean":"2023-10-03T18:31:39.299994+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:31:39.299994+0000","last_undegraded":"2023-10-03T18:31:39.299994+0000","last_fullsized":"2023-10-03T18:31:39.299994+0000","mapping_epoch":12,"log_start":"397'195096","ondisk_log_start":"397'195096","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10001,"log_dups_size":0,"ondisk_log_size":10001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":256,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":24,"snaptrim_duration":1.486569314,"stat_sum":{"num_bytes":2902201159,"num_objects":10514,"num_object_clones":9874,"num_object_copies":21028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10514,"num_whiteouts":0,"num_read":181715,"num_read_kb":659794,"num_write":164914,"num_write_kb":657433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1da","length":"6"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24101627153,"num_objects":72842,"num_object_clones":67578,"num_object_copies":145684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72842,"num_whiteouts":0,"num_read":1526588,"num_read_kb":5506698,"num_write":1373170,"num_write_kb":5471557,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11198156800,"data_stored":135622715938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78030,"internal_metadata":0},"log_size":80428,"ondisk_log_size":80428,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739075,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4022132,"kb_used_data":2104108,"kb_used_omap":1375,"kb_used_meta":1916640,"kb_avail":90349708,"statfs":{"total":96636764160,"available":92518100992,"internally_reserved":0,"allocated":2154606592,"data_stored":21771956055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":1962639492},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3,1,0,3,3],"upper_bound":512},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.9770000000000001,"15min":2.3460000000000001},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":9.0340000000000007,"5min":14.26,"15min":15.964},"last":1.103},{"interface":"front","average":{"1min":1.956,"5min":2.0579999999999998,"15min":2.3610000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.3559999999999999,"5min":12.082000000000001,"15min":34.176000000000002},"last":0.24199999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.3530000000000002,"15min":2.4449999999999998},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.638,"5min":13.836,"15min":17.274000000000001},"last":2.0609999999999999},{"interface":"front","average":{"1min":2.359,"5min":2.0430000000000001,"15min":2.3109999999999999},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":13.462999999999999,"5min":13.462999999999999,"15min":13.842000000000001},"last":2.149}]}]},{"osd":0,"up_from":8,"seq":34359739074,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4926260,"kb_used_data":3375340,"kb_used_omap":2774,"kb_used_meta":1548137,"kb_avail":89445580,"statfs":{"total":96636764160,"available":91592273920,"internally_reserved":0,"allocated":3456348160,"data_stored":46040336164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1585292895},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,6,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.847,"5min":3.2639999999999998,"15min":3.016},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.14799999999999999},"max":{"1min":7.8369999999999997,"5min":21.256,"15min":45.886000000000003},"last":4.7380000000000004},{"interface":"front","average":{"1min":4.9530000000000003,"5min":4.0350000000000001,"15min":3.4220000000000002},"min":{"1min":0.34799999999999998,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":20.763000000000002,"5min":20.763000000000002,"15min":45.555},"last":1.583}]},{"osd":2,"last update":"Tue Oct 3 18:30:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":2.0289999999999999,"15min":1.966},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":4.9269999999999996,"5min":11.252000000000001,"15min":13.058},"last":0.56899999999999995},{"interface":"front","average":{"1min":3.0499999999999998,"5min":3.4980000000000002,"15min":2.6099999999999999},"min":{"1min":0.22700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":18.352,"5min":25.77,"15min":25.77},"last":2.3620000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739042,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6707028,"kb_used_data":5470600,"kb_used_omap":4493,"kb_used_meta":1231922,"kb_avail":87664812,"statfs":{"total":96636764160,"available":89768767488,"internally_reserved":0,"allocated":5601894400,"data_stored":67826847446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1261488568},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,1,4,12,7,11],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":4.9630000000000001,"5min":4.4580000000000002,"15min":3.8260000000000001},"min":{"1min":0.27800000000000002,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":23.126000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":4.2480000000000002},{"interface":"front","average":{"1min":4.1379999999999999,"5min":3.5979999999999999,"15min":3.1659999999999999},"min":{"1min":0.25900000000000001,"5min":0.21199999999999999,"15min":0.19500000000000001},"max":{"1min":22.905000000000001,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:30:40 2023","interfaces":[{"interface":"back","average":{"1min":2.641,"5min":2.956,"15min":2.3879999999999999},"min":{"1min":0.23300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":12.904999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.51800000000000002},{"interface":"front","average":{"1min":2.1030000000000002,"5min":2.6970000000000001,"15min":2.754},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.3490000000000002,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.60799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3450904576,"data_stored":46041009503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5597421568,"data_stored":67823191313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2149830656,"data_stored":21758515122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:31:43.658 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:43.658 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:43.658 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:43.659 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:31:43.659 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:31:44.266 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:44.264+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:31:44.305 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:44.303+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:31:44.364 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:44.362+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:31:45.009 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.006+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:45.061 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.060+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:45.062 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.060+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:45.132 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.124+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:45.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.133+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:45.235 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.233+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:45.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.279+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:45.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.279+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:45.291 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.289+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:45.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.306+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:45.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.306+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:45.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.306+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:45.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.306+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:45.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.306+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:45.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.323+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:45.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.323+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:45.433 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.428+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:45.478 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.472+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:45.478 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.472+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:45.511 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.510+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:45.512 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.510+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:45.549 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:45.546+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:47.519 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.519+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:47.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.519+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:47.523 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.521+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:47.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.573+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:47.581 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.574+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:47.629 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.628+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:47.666 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.663+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:47.688 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.688+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:47.689 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.688+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:47.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.757+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:47.764 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.757+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:47.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.848+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:47.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.849+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:47.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.911+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:47.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.912+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:47.913 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.912+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:47.924 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.920+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:47.949 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.949+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:47.986 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.985+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:47.986 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:47.985+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:48.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:48.000+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:48.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:48.059+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:48.065 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:48.059+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:51.224 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.223+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:51.275 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.271+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:51.307 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.306+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:51.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.337+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:51.338 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.337+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:51.406 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.406+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:51.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.412+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:51.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:51.412+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:53.056 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.054+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:53.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.157+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:53.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.157+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:53.158 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.157+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:53.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.308+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:53.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.308+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:53.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.308+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:53.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.308+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:53.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.324+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:53.646 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:53.639+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:31:53.648 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:53.647+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:31:53.648 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:53.647+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:31:53.982 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.977+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:53.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.993+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:53.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.993+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:53.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:53.993+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:54.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:54.015+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:54.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:54.099+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:55.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.716+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:55.719 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.718+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:55.809 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.808+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:55.826 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.825+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:55.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.863+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:55.890 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.888+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:55.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.929+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:55.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:55.929+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:56.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.007+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:56.010 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.007+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:56.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.011+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:56.528 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.527+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:56.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.594+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:56.612 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.604+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:56.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.720+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:56.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.748+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:56.749 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.748+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:56.905 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.901+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:56.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.907+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:56.908 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.907+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:56.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.921+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:56.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.928+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:56.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.928+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:56.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.928+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:56.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.942+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:56.943 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.942+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:56.950 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.949+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.949+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.950+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.950+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:56.951 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:56.950+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:57.682 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:31:57.675+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:31:57.873 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:57.868+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:57.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:57.977+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:57.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:57.977+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:58.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.017+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:58.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.017+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:58.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.017+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:58.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.017+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:58.020 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.017+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:58.116 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.113+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:58.148 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.145+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:58.170 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.163+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:58.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.179+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:58.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.188+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:58.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.188+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:58.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.189+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:58.193 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.189+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:58.207 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:58.206+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:59.619 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.619+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:59.632 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.631+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:59.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.664+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:59.671 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.664+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:59.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.727+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:59.728 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.727+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:59.735 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.734+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:31:59.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.758+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:31:59.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.785+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:31:59.787 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.785+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:31:59.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.851+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:31:59.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.851+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:31:59.853 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.851+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:31:59.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.946+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:31:59.947 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.947+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:31:59.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.948+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:31:59.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.948+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:31:59.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:31:59.948+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:00.017 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.016+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:00.067 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.058+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:00.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.145+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:00.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.153+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:00.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.190+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:00.191 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.190+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:00.292 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.286+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:00.293 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.286+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:00.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.330+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:00.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.330+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:00.346 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.343+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:00.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.350+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:00.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:00.385+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:00.678 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:00.677+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:32:00.702 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:00.694+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:32:02.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.672+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:02.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.778+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:02.883 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.882+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:02.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.893+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:02.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.893+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:02.895 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.893+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:02.896 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.893+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:02.980 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:02.976+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:03.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.000+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:03.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.245+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:03.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.263+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:03.268 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.263+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:03.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.281+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:03.283 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.282+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:03.351 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:03.350+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:03.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:32:04.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:32:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:32:05.534 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2219,"stamp":"2023-10-03T18:32:03.762826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24131795038,"num_objects":72577,"num_object_clones":67305,"num_object_copies":145154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72577,"num_whiteouts":0,"num_read":1535179,"num_read_kb":5536325,"num_write":1380422,"num_write_kb":5501754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80599,"ondisk_log_size":80599,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":15940928,"kb_used_data":11006544,"kb_used_omap":8643,"kb_used_meta":4925692,"kb_avail":267174592,"statfs":{"total":289910292480,"available":273586782208,"internally_reserved":0,"allocated":11270701056,"data_stored":135307328464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8851301,"internal_metadata":5043908763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4,5,7,24,14,5],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7835949,"num_objects":-140,"num_object_clones":-146,"num_object_copies":-280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-140,"num_whiteouts":0,"num_read":3148,"num_read_kb":8328,"num_write":1913,"num_write_kb":7495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.256055"},"pg_stats":[{"pgid":"2.7","version":"446'211250","reported_seq":368110,"reported_epoch":446,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:01.205856+0000","last_change":"2023-10-03T18:32:00.594747+0000","last_active":"2023-10-03T18:32:01.205856+0000","last_peered":"2023-10-03T18:32:01.205856+0000","last_clean":"2023-10-03T18:32:01.205856+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:32:01.205856+0000","last_undegraded":"2023-10-03T18:32:01.205856+0000","last_fullsized":"2023-10-03T18:32:01.205856+0000","mapping_epoch":12,"log_start":"403'201215","ondisk_log_start":"403'201215","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":332,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:07:37.550799+0000","scrub_duration":505.38444223300002,"objects_trimmed":6,"snaptrim_duration":4.1680224890000002,"stat_sum":{"num_bytes":3005882803,"num_objects":13284,"num_object_clones":12634,"num_object_copies":26568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13284,"num_whiteouts":0,"num_read":186768,"num_read_kb":688273,"num_write":171451,"num_write_kb":683641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"2"}]},{"pgid":"2.6","version":"446'195379","reported_seq":341532,"reported_epoch":446,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:01.172034+0000","last_change":"2023-10-03T18:31:58.578359+0000","last_active":"2023-10-03T18:32:01.172034+0000","last_peered":"2023-10-03T18:32:01.172034+0000","last_clean":"2023-10-03T18:32:01.172034+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:32:01.172034+0000","last_undegraded":"2023-10-03T18:32:01.172034+0000","last_fullsized":"2023-10-03T18:32:01.172034+0000","mapping_epoch":12,"log_start":"399'185288","ondisk_log_start":"399'185288","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10091,"log_dups_size":0,"ondisk_log_size":10091,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":245,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":5.7992806029999997,"stat_sum":{"num_bytes":2776855199,"num_objects":9830,"num_object_clones":9225,"num_object_copies":19660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9830,"num_whiteouts":0,"num_read":173369,"num_read_kb":635460,"num_write":158316,"num_write_kb":631800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"5"}]},{"pgid":"2.5","version":"446'234479","reported_seq":406842,"reported_epoch":446,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:01.169770+0000","last_change":"2023-10-03T18:32:00.947975+0000","last_active":"2023-10-03T18:32:01.169770+0000","last_peered":"2023-10-03T18:32:01.169770+0000","last_clean":"2023-10-03T18:32:01.169770+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:32:01.169770+0000","last_undegraded":"2023-10-03T18:32:01.169770+0000","last_fullsized":"2023-10-03T18:32:01.169770+0000","mapping_epoch":12,"log_start":"411'224391","ondisk_log_start":"411'224391","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10088,"log_dups_size":0,"ondisk_log_size":10088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":280,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.076126017000000004,"stat_sum":{"num_bytes":3258139329,"num_objects":12686,"num_object_clones":11975,"num_object_copies":25372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12686,"num_whiteouts":0,"num_read":214533,"num_read_kb":753144,"num_write":186877,"num_write_kb":743548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"}]},{"pgid":"2.0","version":"446'233116","reported_seq":394619,"reported_epoch":446,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:01.417958+0000","last_change":"2023-10-03T18:31:59.842876+0000","last_active":"2023-10-03T18:32:01.417958+0000","last_peered":"2023-10-03T18:32:01.417958+0000","last_clean":"2023-10-03T18:32:01.417958+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:32:01.417958+0000","last_undegraded":"2023-10-03T18:32:01.417958+0000","last_fullsized":"2023-10-03T18:32:01.417958+0000","mapping_epoch":12,"log_start":"416'223026","ondisk_log_start":"416'223026","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10090,"log_dups_size":0,"ondisk_log_size":10090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":152,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T18:43:52.897139+0000","scrub_duration":734.24538164700004,"objects_trimmed":8,"snaptrim_duration":7.10515148,"stat_sum":{"num_bytes":3147518710,"num_objects":8112,"num_object_clones":7425,"num_object_copies":16224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8112,"num_whiteouts":0,"num_read":194345,"num_read_kb":717684,"num_write":180237,"num_write_kb":718757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"8"}]},{"pgid":"1.0","version":"88'99","reported_seq":613,"reported_epoch":446,"state":"active+clean","last_fresh":"2023-10-03T18:32:00.895815+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:32:00.895815+0000","last_peered":"2023-10-03T18:32:00.895815+0000","last_clean":"2023-10-03T18:32:00.895815+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:32:00.895815+0000","last_undegraded":"2023-10-03T18:32:00.895815+0000","last_fullsized":"2023-10-03T18:32:00.895815+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"446'225560","reported_seq":390735,"reported_epoch":446,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:01.100573+0000","last_change":"2023-10-03T18:32:00.566959+0000","last_active":"2023-10-03T18:32:01.100573+0000","last_peered":"2023-10-03T18:32:01.100573+0000","last_clean":"2023-10-03T18:32:01.100573+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:32:01.100573+0000","last_undegraded":"2023-10-03T18:32:01.100573+0000","last_fullsized":"2023-10-03T18:32:01.100573+0000","mapping_epoch":12,"log_start":"410'215513","ondisk_log_start":"410'215513","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10047,"log_dups_size":0,"ondisk_log_size":10047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":270,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":7.8009680479999997,"stat_sum":{"num_bytes":3200570085,"num_objects":12144,"num_object_clones":11447,"num_object_copies":24288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12144,"num_whiteouts":0,"num_read":197949,"num_read_kb":730039,"num_write":182601,"num_write_kb":728143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"}]},{"pgid":"2.1","version":"446'223402","reported_seq":372910,"reported_epoch":446,"state":"active+clean","last_fresh":"2023-10-03T18:32:01.373993+0000","last_change":"2023-10-03T18:31:44.475665+0000","last_active":"2023-10-03T18:32:01.373993+0000","last_peered":"2023-10-03T18:32:01.373993+0000","last_clean":"2023-10-03T18:32:01.373993+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:32:01.373993+0000","last_undegraded":"2023-10-03T18:32:01.373993+0000","last_fullsized":"2023-10-03T18:32:01.373993+0000","mapping_epoch":12,"log_start":"411'213330","ondisk_log_start":"411'213330","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T04:52:33.696720+0000","scrub_duration":424.08306342600002,"objects_trimmed":100,"snaptrim_duration":6.7315022610000002,"stat_sum":{"num_bytes":2916741692,"num_objects":2988,"num_object_clones":2354,"num_object_copies":5976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2988,"num_whiteouts":0,"num_read":186992,"num_read_kb":667369,"num_write":166861,"num_write_kb":664493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"}]},{"pgid":"2.2","version":"447'223479","reported_seq":379028,"reported_epoch":447,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:02.773781+0000","last_change":"2023-10-03T18:32:01.840798+0000","last_active":"2023-10-03T18:32:02.773781+0000","last_peered":"2023-10-03T18:32:02.773781+0000","last_clean":"2023-10-03T18:32:02.773781+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:32:02.773781+0000","last_undegraded":"2023-10-03T18:32:02.773781+0000","last_fullsized":"2023-10-03T18:32:02.773781+0000","mapping_epoch":12,"log_start":"411'213467","ondisk_log_start":"411'213467","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:22:55.893326+0000","scrub_duration":370.574046178,"objects_trimmed":16,"snaptrim_duration":8.1805440090000001,"stat_sum":{"num_bytes":2919342953,"num_objects":3002,"num_object_clones":2358,"num_object_copies":6004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3002,"num_whiteouts":0,"num_read":198486,"num_read_kb":681102,"num_write":168081,"num_write_kb":668272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"}]},{"pgid":"2.4","version":"446'206365","reported_seq":358680,"reported_epoch":446,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:01.187478+0000","last_change":"2023-10-03T18:32:01.068195+0000","last_active":"2023-10-03T18:32:01.187478+0000","last_peered":"2023-10-03T18:32:01.187478+0000","last_clean":"2023-10-03T18:32:01.187478+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:32:01.187478+0000","last_undegraded":"2023-10-03T18:32:01.187478+0000","last_fullsized":"2023-10-03T18:32:01.187478+0000","mapping_epoch":12,"log_start":"402'196300","ondisk_log_start":"402'196300","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":253,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":4.2160354299999998,"stat_sum":{"num_bytes":2906153899,"num_objects":10529,"num_object_clones":9887,"num_object_copies":21058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10529,"num_whiteouts":0,"num_read":182631,"num_read_kb":663163,"num_write":165861,"num_write_kb":661206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"4"},{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"7"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24131204670,"num_objects":72575,"num_object_clones":67305,"num_object_copies":145150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72575,"num_whiteouts":0,"num_read":1535073,"num_read_kb":5536234,"num_write":1380285,"num_write_kb":5499860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11255255040,"data_stored":135308127155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78030,"internal_metadata":0},"log_size":80500,"ondisk_log_size":80500,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739079,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4076916,"kb_used_data":2114536,"kb_used_omap":1375,"kb_used_meta":1960992,"kb_avail":90294924,"statfs":{"total":96636764160,"available":92462002176,"internally_reserved":0,"allocated":2165284864,"data_stored":21583934497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":2008055940},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,2,3,2,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.9770000000000001,"15min":2.3460000000000001},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":9.0340000000000007,"5min":14.26,"15min":15.964},"last":4.1040000000000001},{"interface":"front","average":{"1min":1.956,"5min":2.0579999999999998,"15min":2.3610000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.3559999999999999,"5min":12.082000000000001,"15min":34.176000000000002},"last":9.7829999999999995}]},{"osd":1,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.3530000000000002,"15min":2.4449999999999998},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.638,"5min":13.836,"15min":17.274000000000001},"last":7.8449999999999998},{"interface":"front","average":{"1min":2.359,"5min":2.0430000000000001,"15min":2.3109999999999999},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":13.462999999999999,"5min":13.462999999999999,"15min":13.842000000000001},"last":1.1459999999999999}]}]},{"osd":0,"up_from":8,"seq":34359739078,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4992696,"kb_used_data":3391780,"kb_used_omap":2774,"kb_used_meta":1598121,"kb_avail":89379144,"statfs":{"total":96636764160,"available":91524243456,"internally_reserved":0,"allocated":3473182720,"data_stored":46080656399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1636476511},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":3.3290000000000002,"5min":3.4209999999999998,"15min":3.0630000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":11.391,"5min":21.256,"15min":45.886000000000003},"last":2.375},{"interface":"front","average":{"1min":2.9049999999999998,"5min":4.0919999999999996,"15min":3.4359999999999999},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":17.323,"5min":20.763000000000002,"15min":45.555},"last":2.351}]},{"osd":2,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":2.335,"5min":1.9990000000000001,"15min":1.946},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":8.1519999999999992,"5min":8.9860000000000007,"15min":13.058},"last":0.20899999999999999},{"interface":"front","average":{"1min":3.0859999999999999,"5min":3.4009999999999998,"15min":2.7160000000000002},"min":{"1min":0.25700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":14.611000000000001,"5min":25.77,"15min":25.77},"last":1.885}]}]},{"osd":1,"up_from":8,"seq":34359739046,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6871316,"kb_used_data":5500228,"kb_used_omap":4493,"kb_used_meta":1366578,"kb_avail":87500524,"statfs":{"total":96636764160,"available":89600536576,"internally_reserved":0,"allocated":5632233472,"data_stored":67642737568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1399376312},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3,4,22,14,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:31:40 2023","interfaces":[{"interface":"back","average":{"1min":4.7999999999999998,"5min":4.9649999999999999,"15min":3.96},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":31.344000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":10.069000000000001},{"interface":"front","average":{"1min":3.0960000000000001,"5min":3.7370000000000001,"15min":3.1309999999999998},"min":{"1min":0.30499999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":22.692,"5min":52.982999999999997,"15min":52.982999999999997},"last":2.2490000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:31:40 2023","interfaces":[{"interface":"back","average":{"1min":2.379,"5min":2.8759999999999999,"15min":2.4710000000000001},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.247999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":1.347},{"interface":"front","average":{"1min":3.0830000000000002,"5min":2.9729999999999999,"15min":2.694},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":14.926,"5min":36.332999999999998,"15min":36.332999999999998},"last":2.294}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3467620352,"data_stored":46088121594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5627080704,"data_stored":67636250274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2160553984,"data_stored":21583755287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:32:05.535 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:05.535 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:05.536 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:05.536 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:05.536 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:32:07.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.209+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:07.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.261+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:07.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.261+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:07.311 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.309+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:07.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.470+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:07.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.470+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:07.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.758+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:07.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.758+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:07.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:07.791+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:17.318 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.314+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:17.395 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.392+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:17.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.421+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:17.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.609+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:17.656 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.654+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:17.684 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:17.681+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:32:17.783 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.781+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:17.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.853+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:17.856 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.856+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:17.876 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.874+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:17.877 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.874+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:17.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.893+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:17.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.893+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:17.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.893+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:17.894 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:17.893+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:18.036 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:18.035+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:22.121 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:32:22.119+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:32:22.121 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:32:22.119+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:32:22.121 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:32:22.119+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:32:22.121 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:32:22.119+0000 7f0623ebd700 -1 reset not still connected to 0x55870e449790 2023-10-03T18:32:22.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:22.785+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:22.786 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:22.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:22.787 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:22.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:22.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:22.957+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:23.075 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.073+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:23.078 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.077+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:23.078 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.077+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:23.089 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.088+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:23.114 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.113+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:23.114 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:23.113+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:24.534 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.533+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:24.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.540+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:24.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:24.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.672+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:24.739 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.732+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:24.740 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.732+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:24.848 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.846+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:24.849 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.847+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:24.870 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.870+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:24.957 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:24.955+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:25.007 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:25.007+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:25.537 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:32:26.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:25.998+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:26.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:26.159+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:26.163 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:26.159+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:26.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:26.169+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:26.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:26.193+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:26.217 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:26.217+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:26.315 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:32:26.332 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:32:26.350 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2236,"stamp":"2023-10-03T18:32:25.516278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24152348465,"num_objects":74240,"num_object_clones":68974,"num_object_copies":148480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74240,"num_whiteouts":0,"num_read":1543522,"num_read_kb":5559122,"num_write":1385509,"num_write_kb":5521539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80556,"ondisk_log_size":80556,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16357628,"kb_used_data":11045344,"kb_used_omap":8643,"kb_used_meta":5303612,"kb_avail":266757892,"statfs":{"total":289910292480,"available":273160081408,"internally_reserved":0,"allocated":11310432256,"data_stored":139113404475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8851301,"internal_metadata":5430898843},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,2,2,8,9,12,20,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9449472,"num_objects":683,"num_object_clones":683,"num_object_copies":1366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":683,"num_whiteouts":0,"num_read":2365,"num_read_kb":9400,"num_write":2310,"num_write_kb":9240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003071"},"pg_stats":[{"pgid":"2.7","version":"455'212534","reported_seq":369805,"reported_epoch":455,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:24.408492+0000","last_change":"2023-10-03T18:32:23.871124+0000","last_active":"2023-10-03T18:32:24.408492+0000","last_peered":"2023-10-03T18:32:24.408492+0000","last_clean":"2023-10-03T18:32:24.408492+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:32:24.408492+0000","last_undegraded":"2023-10-03T18:32:24.408492+0000","last_fullsized":"2023-10-03T18:32:24.408492+0000","mapping_epoch":12,"log_start":"412'202521","ondisk_log_start":"412'202521","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":348,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:05:52.979597+0000","scrub_duration":505.38444223300002,"objects_trimmed":8,"snaptrim_duration":1.051854168,"stat_sum":{"num_bytes":3008500146,"num_objects":13529,"num_object_clones":12879,"num_object_copies":27058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13529,"num_whiteouts":0,"num_read":187849,"num_read_kb":691424,"num_write":172107,"num_write_kb":686209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e9","length":"1"}]},{"pgid":"2.6","version":"455'196494","reported_seq":343033,"reported_epoch":455,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:24.379807+0000","last_change":"2023-10-03T18:32:22.554001+0000","last_active":"2023-10-03T18:32:24.379807+0000","last_peered":"2023-10-03T18:32:24.379807+0000","last_clean":"2023-10-03T18:32:24.379807+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:32:24.379807+0000","last_undegraded":"2023-10-03T18:32:24.379807+0000","last_fullsized":"2023-10-03T18:32:24.379807+0000","mapping_epoch":12,"log_start":"405'186395","ondisk_log_start":"405'186395","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10099,"log_dups_size":0,"ondisk_log_size":10099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":262,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.7552615879999998,"stat_sum":{"num_bytes":2779505313,"num_objects":10086,"num_object_clones":9481,"num_object_copies":20172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10086,"num_whiteouts":0,"num_read":174209,"num_read_kb":638177,"num_write":158910,"num_write_kb":634139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e9","length":"1"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"}]},{"pgid":"2.5","version":"455'235770","reported_seq":408602,"reported_epoch":455,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:24.313852+0000","last_change":"2023-10-03T18:32:21.439983+0000","last_active":"2023-10-03T18:32:24.313852+0000","last_peered":"2023-10-03T18:32:24.313852+0000","last_clean":"2023-10-03T18:32:24.313852+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:32:24.313852+0000","last_undegraded":"2023-10-03T18:32:24.313852+0000","last_fullsized":"2023-10-03T18:32:24.313852+0000","mapping_epoch":12,"log_start":"414'225698","ondisk_log_start":"414'225698","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":298,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":3.6395477870000001,"stat_sum":{"num_bytes":3260973558,"num_objects":13002,"num_object_clones":12295,"num_object_copies":26004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13002,"num_whiteouts":0,"num_read":215973,"num_read_kb":756530,"num_write":187609,"num_write_kb":746344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e9","length":"1"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"2"}]},{"pgid":"2.0","version":"454'234601","reported_seq":396317,"reported_epoch":454,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:21.534190+0000","last_change":"2023-10-03T18:32:19.396774+0000","last_active":"2023-10-03T18:32:21.534190+0000","last_peered":"2023-10-03T18:32:21.534190+0000","last_clean":"2023-10-03T18:32:21.534190+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:32:21.534190+0000","last_undegraded":"2023-10-03T18:32:21.534190+0000","last_fullsized":"2023-10-03T18:32:21.534190+0000","mapping_epoch":12,"log_start":"425'224538","ondisk_log_start":"425'224538","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":165,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:28:43.415540+0000","scrub_duration":734.24538164700004,"objects_trimmed":53,"snaptrim_duration":1.6417030969999999,"stat_sum":{"num_bytes":3150132009,"num_objects":8230,"num_object_clones":7542,"num_object_copies":16460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8230,"num_whiteouts":0,"num_read":195264,"num_read_kb":720203,"num_write":180879,"num_write_kb":721259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":622,"reported_epoch":454,"state":"active+clean","last_fresh":"2023-10-03T18:32:17.893293+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:32:17.893293+0000","last_peered":"2023-10-03T18:32:17.893293+0000","last_clean":"2023-10-03T18:32:17.893293+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:32:17.893293+0000","last_undegraded":"2023-10-03T18:32:17.893293+0000","last_fullsized":"2023-10-03T18:32:17.893293+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"455'226797","reported_seq":392344,"reported_epoch":455,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:24.211855+0000","last_change":"2023-10-03T18:32:22.576618+0000","last_active":"2023-10-03T18:32:24.211855+0000","last_peered":"2023-10-03T18:32:24.211855+0000","last_clean":"2023-10-03T18:32:24.211855+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:32:24.211855+0000","last_undegraded":"2023-10-03T18:32:24.211855+0000","last_fullsized":"2023-10-03T18:32:24.211855+0000","mapping_epoch":12,"log_start":"413'216716","ondisk_log_start":"413'216716","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10081,"log_dups_size":0,"ondisk_log_size":10081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":288,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":12,"snaptrim_duration":4.6250615479999997,"stat_sum":{"num_bytes":3203248869,"num_objects":12451,"num_object_clones":11754,"num_object_copies":24902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12451,"num_whiteouts":0,"num_read":198892,"num_read_kb":733040,"num_write":183269,"num_write_kb":730768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e9","length":"1"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"}]},{"pgid":"2.1","version":"454'224822","reported_seq":374611,"reported_epoch":454,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:21.541466+0000","last_change":"2023-10-03T18:32:20.074001+0000","last_active":"2023-10-03T18:32:21.541466+0000","last_peered":"2023-10-03T18:32:21.541466+0000","last_clean":"2023-10-03T18:32:21.541466+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:32:21.541466+0000","last_undegraded":"2023-10-03T18:32:21.541466+0000","last_fullsized":"2023-10-03T18:32:21.541466+0000","mapping_epoch":12,"log_start":"414'214735","ondisk_log_start":"414'214735","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10087,"log_dups_size":0,"ondisk_log_size":10087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:00:00.281706+0000","scrub_duration":424.08306342600002,"objects_trimmed":46,"snaptrim_duration":2.2838177270000002,"stat_sum":{"num_bytes":2919043644,"num_objects":3066,"num_object_clones":2432,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":188126,"num_read_kb":670090,"num_write":167429,"num_write_kb":666680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1e8","length":"2"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"8"}]},{"pgid":"2.2","version":"455'224894","reported_seq":380697,"reported_epoch":455,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:22.920372+0000","last_change":"2023-10-03T18:32:22.880230+0000","last_active":"2023-10-03T18:32:22.920372+0000","last_peered":"2023-10-03T18:32:22.920372+0000","last_clean":"2023-10-03T18:32:22.920372+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:32:22.920372+0000","last_undegraded":"2023-10-03T18:32:22.920372+0000","last_fullsized":"2023-10-03T18:32:22.920372+0000","mapping_epoch":12,"log_start":"414'214881","ondisk_log_start":"414'214881","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:32:44.050089+0000","scrub_duration":370.574046178,"objects_trimmed":2,"snaptrim_duration":0.053229017000000003,"stat_sum":{"num_bytes":2921521974,"num_objects":3078,"num_object_clones":2435,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3078,"num_whiteouts":0,"num_read":199449,"num_read_kb":683619,"num_write":168633,"num_write_kb":670418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e9","length":"1"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"9"}]},{"pgid":"2.4","version":"455'207635","reported_seq":360302,"reported_epoch":455,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:24.397840+0000","last_change":"2023-10-03T18:32:23.871293+0000","last_active":"2023-10-03T18:32:24.397840+0000","last_peered":"2023-10-03T18:32:24.397840+0000","last_clean":"2023-10-03T18:32:24.397840+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:32:24.397840+0000","last_undegraded":"2023-10-03T18:32:24.397840+0000","last_fullsized":"2023-10-03T18:32:24.397840+0000","mapping_epoch":12,"log_start":"411'197606","ondisk_log_start":"411'197606","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10029,"log_dups_size":0,"ondisk_log_size":10029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":270,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":2.7234606289999999,"stat_sum":{"num_bytes":2908832584,"num_objects":10796,"num_object_clones":10156,"num_object_copies":21592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10796,"num_whiteouts":0,"num_read":183654,"num_read_kb":665948,"num_write":166536,"num_write_kb":663828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e9","length":"1"},{"start":"1ec","length":"5"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24151758097,"num_objects":74238,"num_object_clones":68974,"num_object_copies":148476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74238,"num_whiteouts":0,"num_read":1543416,"num_read_kb":5559031,"num_write":1385372,"num_write_kb":5519645,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11294797824,"data_stored":139159910946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78030,"internal_metadata":0},"log_size":80457,"ondisk_log_size":80457,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739083,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4131716,"kb_used_data":2121532,"kb_used_omap":1375,"kb_used_meta":2008800,"kb_avail":90240124,"statfs":{"total":96636764160,"available":92405886976,"internally_reserved":0,"allocated":2172448768,"data_stored":22152623101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1408892,"internal_metadata":2057011332},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,2,1,7],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":1.873,"5min":1.9770000000000001,"15min":2.3460000000000001},"min":{"1min":0.19,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":9.0340000000000007,"5min":14.26,"15min":15.964},"last":0.26200000000000001},{"interface":"front","average":{"1min":1.956,"5min":2.0579999999999998,"15min":2.3610000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.3559999999999999,"5min":12.082000000000001,"15min":34.176000000000002},"last":0.75900000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:31:30 2023","interfaces":[{"interface":"back","average":{"1min":2.5270000000000001,"5min":2.3530000000000002,"15min":2.4449999999999998},"min":{"1min":0.32100000000000001,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":11.638,"5min":13.836,"15min":17.274000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":2.359,"5min":2.0430000000000001,"15min":2.3109999999999999},"min":{"1min":0.23499999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":13.462999999999999,"5min":13.462999999999999,"15min":13.842000000000001},"last":2.2970000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739082,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5125736,"kb_used_data":3402720,"kb_used_omap":2774,"kb_used_meta":1720233,"kb_avail":89246104,"statfs":{"total":96636764160,"available":91388010496,"internally_reserved":0,"allocated":3484385280,"data_stored":47355173530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1761519199},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1,2,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":3.3290000000000002,"5min":3.4209999999999998,"15min":3.0630000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":11.391,"5min":21.256,"15min":45.886000000000003},"last":2.6190000000000002},{"interface":"front","average":{"1min":2.9049999999999998,"5min":4.0919999999999996,"15min":3.4359999999999999},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":17.323,"5min":20.763000000000002,"15min":45.555},"last":3.0870000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":2.335,"5min":1.9990000000000001,"15min":1.946},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":8.1519999999999992,"5min":8.9860000000000007,"15min":13.058},"last":0.27900000000000003},{"interface":"front","average":{"1min":3.0859999999999999,"5min":3.4009999999999998,"15min":2.7160000000000002},"min":{"1min":0.25700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":14.611000000000001,"5min":25.77,"15min":25.77},"last":0.247}]}]},{"osd":1,"up_from":8,"seq":34359739050,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7100176,"kb_used_data":5521092,"kb_used_omap":4493,"kb_used_meta":1574578,"kb_avail":87271664,"statfs":{"total":96636764160,"available":89366183936,"internally_reserved":0,"allocated":5653598208,"data_stored":69605607844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1612368312},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1,6,5,11,9,7],"upper_bound":512},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:31:40 2023","interfaces":[{"interface":"back","average":{"1min":4.7999999999999998,"5min":4.9649999999999999,"15min":3.96},"min":{"1min":0.22600000000000001,"5min":0.22,"15min":0.17399999999999999},"max":{"1min":31.344000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":3.0960000000000001,"5min":3.7370000000000001,"15min":3.1309999999999998},"min":{"1min":0.30499999999999999,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":22.692,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.60399999999999998}]},{"osd":2,"last update":"Tue Oct 3 18:31:40 2023","interfaces":[{"interface":"back","average":{"1min":2.379,"5min":2.8759999999999999,"15min":2.4710000000000001},"min":{"1min":0.224,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":14.247999999999999,"5min":34.395000000000003,"15min":34.395000000000003},"last":0.24199999999999999},{"interface":"front","average":{"1min":3.0830000000000002,"5min":2.9729999999999999,"15min":2.694},"min":{"1min":0.25,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":14.926,"5min":36.332999999999998,"15min":36.332999999999998},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3478781952,"data_stored":47364115655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5648392192,"data_stored":69623002385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2167623680,"data_stored":22172792906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10195,"internal_metadata":0}]}} 2023-10-03T18:32:26.352 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:26.352 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:26.353 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:26.353 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:26.353 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:32:26.538 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:26.537+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:32:26.583 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:26.580+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:32:26.648 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:26.647+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:32:31.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:31.288+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:31.290 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:31.288+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:31.336 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:31.335+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:33.111 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.109+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:33.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.152+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:33.198 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.196+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:33.239 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.238+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:33.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.261+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:33.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.261+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:33.267 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.261+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:33.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.383+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:33.576 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:33.570+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:34.016 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.014+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:34.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.058+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:34.060 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.058+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:34.096 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.091+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:34.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.118+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:34.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.118+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:34.557 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.554+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:34.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.560+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:34.567 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.565+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:34.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.615+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:34.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.633+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:34.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.633+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:34.673 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.671+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:34.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.671+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:34.674 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:34.671+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:35.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.364+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:32:35.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.364+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:35.366 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.364+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:35.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.443+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:35.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.443+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:35.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.454+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:35.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.469+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:35.470 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.469+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:35.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.469+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:35.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.469+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:35.471 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:35.469+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:36.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.451+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:36.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.451+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:36.454 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.451+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:36.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.457+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:36.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.457+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:36.459 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.457+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:36.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:36.457+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:38.135 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.124+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:38.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.140+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:38.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.140+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:38.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.140+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:38.146 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.140+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:38.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.159+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:38.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:38.159+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:39.920 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:32:39.916+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:32:43.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.475+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:43.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.477+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:43.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.477+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:43.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.477+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:43.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.482+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:43.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.518+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:43.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.518+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:43.589 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.588+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:43.590 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.588+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:43.601 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:43.596+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:46.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:32:47.456 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.448+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:47.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.491+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:47.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.516+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:47.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.517+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:47.556 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:47.554+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:47.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:32:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:32:47.599 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2250,"stamp":"2023-10-03T18:32:47.531682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24177415985,"num_objects":74292,"num_object_clones":69026,"num_object_copies":148584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74292,"num_whiteouts":0,"num_read":1549900,"num_read_kb":5584482,"num_write":1391572,"num_write_kb":5545791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80433,"ondisk_log_size":80433,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16704120,"kb_used_data":11091104,"kb_used_omap":8762,"kb_used_meta":5604229,"kb_avail":266411400,"statfs":{"total":289910292480,"available":272805273600,"internally_reserved":0,"allocated":11357290496,"data_stored":139707001659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8973241,"internal_metadata":5738730567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,4,3,4,20,7,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12623872,"num_objects":-89,"num_object_clones":-89,"num_object_copies":-178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-89,"num_whiteouts":0,"num_read":3262,"num_read_kb":12960,"num_write":3044,"num_write_kb":12176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.759265"},"pg_stats":[{"pgid":"2.7","version":"458'213726","reported_seq":371564,"reported_epoch":458,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:46.071060+0000","last_change":"2023-10-03T18:32:44.382681+0000","last_active":"2023-10-03T18:32:46.071060+0000","last_peered":"2023-10-03T18:32:46.071060+0000","last_clean":"2023-10-03T18:32:46.071060+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:32:46.071060+0000","last_undegraded":"2023-10-03T18:32:46.071060+0000","last_fullsized":"2023-10-03T18:32:46.071060+0000","mapping_epoch":12,"log_start":"415'203723","ondisk_log_start":"415'203723","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10003,"log_dups_size":0,"ondisk_log_size":10003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":345,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:05:52.979597+0000","scrub_duration":505.38444223300002,"objects_trimmed":20,"snaptrim_duration":1.5841242179999999,"stat_sum":{"num_bytes":3011719602,"num_objects":13620,"num_object_clones":12970,"num_object_copies":27240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13620,"num_whiteouts":0,"num_read":188629,"num_read_kb":694524,"num_write":172893,"num_write_kb":689353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ee","length":"1"}]},{"pgid":"2.6","version":"458'197566","reported_seq":344645,"reported_epoch":458,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:46.095689+0000","last_change":"2023-10-03T18:32:43.233522+0000","last_active":"2023-10-03T18:32:46.095689+0000","last_peered":"2023-10-03T18:32:46.095689+0000","last_clean":"2023-10-03T18:32:46.095689+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:32:46.095689+0000","last_undegraded":"2023-10-03T18:32:46.095689+0000","last_fullsized":"2023-10-03T18:32:46.095689+0000","mapping_epoch":12,"log_start":"412'187499","ondisk_log_start":"412'187499","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":259,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0.42969921100000003,"stat_sum":{"num_bytes":2782352033,"num_objects":10157,"num_object_clones":9552,"num_object_copies":20314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10157,"num_whiteouts":0,"num_read":174947,"num_read_kb":641121,"num_write":159604,"num_write_kb":636915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"1"}]},{"pgid":"2.5","version":"458'237040","reported_seq":410470,"reported_epoch":458,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:46.091803+0000","last_change":"2023-10-03T18:32:45.756708+0000","last_active":"2023-10-03T18:32:46.091803+0000","last_peered":"2023-10-03T18:32:46.091803+0000","last_clean":"2023-10-03T18:32:46.091803+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:32:46.091803+0000","last_undegraded":"2023-10-03T18:32:46.091803+0000","last_fullsized":"2023-10-03T18:32:46.091803+0000","mapping_epoch":12,"log_start":"417'227005","ondisk_log_start":"417'227005","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":295,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":4.326200021,"stat_sum":{"num_bytes":3264172534,"num_objects":13097,"num_object_clones":12390,"num_object_copies":26194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13097,"num_whiteouts":0,"num_read":216814,"num_read_kb":759878,"num_write":188388,"num_write_kb":749460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"5"}]},{"pgid":"2.0","version":"458'236459","reported_seq":398807,"reported_epoch":458,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:46.782715+0000","last_change":"2023-10-03T18:32:46.307340+0000","last_active":"2023-10-03T18:32:46.782715+0000","last_peered":"2023-10-03T18:32:46.782715+0000","last_clean":"2023-10-03T18:32:46.782715+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:32:46.782715+0000","last_undegraded":"2023-10-03T18:32:46.782715+0000","last_fullsized":"2023-10-03T18:32:46.782715+0000","mapping_epoch":12,"log_start":"428'226445","ondisk_log_start":"428'226445","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":153,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:28:43.415540+0000","scrub_duration":734.24538164700004,"objects_trimmed":59,"snaptrim_duration":3.5192687349999998,"stat_sum":{"num_bytes":3153875753,"num_objects":8103,"num_object_clones":7415,"num_object_copies":16206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8103,"num_whiteouts":0,"num_read":196233,"num_read_kb":724051,"num_write":181778,"num_write_kb":724855,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"7"}]},{"pgid":"1.0","version":"88'99","reported_seq":625,"reported_epoch":457,"state":"active+clean","last_fresh":"2023-10-03T18:32:37.841926+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:32:37.841926+0000","last_peered":"2023-10-03T18:32:37.841926+0000","last_clean":"2023-10-03T18:32:37.841926+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:32:37.841926+0000","last_undegraded":"2023-10-03T18:32:37.841926+0000","last_fullsized":"2023-10-03T18:32:37.841926+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"458'227997","reported_seq":394092,"reported_epoch":458,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:46.021120+0000","last_change":"2023-10-03T18:32:44.382951+0000","last_active":"2023-10-03T18:32:46.021120+0000","last_peered":"2023-10-03T18:32:46.021120+0000","last_clean":"2023-10-03T18:32:46.021120+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:32:46.021120+0000","last_undegraded":"2023-10-03T18:32:46.021120+0000","last_fullsized":"2023-10-03T18:32:46.021120+0000","mapping_epoch":12,"log_start":"416'217921","ondisk_log_start":"416'217921","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10076,"log_dups_size":0,"ondisk_log_size":10076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":285,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":28,"snaptrim_duration":3.3746678929999998,"stat_sum":{"num_bytes":3206312677,"num_objects":12542,"num_object_clones":11845,"num_object_copies":25084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12542,"num_whiteouts":0,"num_read":199675,"num_read_kb":736140,"num_write":184021,"num_write_kb":733776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"3"}]},{"pgid":"2.1","version":"458'226518","reported_seq":376863,"reported_epoch":458,"state":"active+clean","last_fresh":"2023-10-03T18:32:46.745785+0000","last_change":"2023-10-03T18:32:42.979111+0000","last_active":"2023-10-03T18:32:46.745785+0000","last_peered":"2023-10-03T18:32:46.745785+0000","last_clean":"2023-10-03T18:32:46.745785+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:32:46.745785+0000","last_undegraded":"2023-10-03T18:32:46.745785+0000","last_fullsized":"2023-10-03T18:32:46.745785+0000","mapping_epoch":12,"log_start":"425'216443","ondisk_log_start":"425'216443","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10075,"log_dups_size":0,"ondisk_log_size":10075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:00:00.281706+0000","scrub_duration":424.08306342600002,"objects_trimmed":2,"snaptrim_duration":0.184502676,"stat_sum":{"num_bytes":2922361404,"num_objects":2942,"num_object_clones":2308,"num_object_copies":5884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2942,"num_whiteouts":0,"num_read":188972,"num_read_kb":673446,"num_write":168242,"num_write_kb":669932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"}]},{"pgid":"2.2","version":"458'226313","reported_seq":382551,"reported_epoch":458,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:32:43.118168+0000","last_change":"2023-10-03T18:32:42.790462+0000","last_active":"2023-10-03T18:32:43.118168+0000","last_peered":"2023-10-03T18:32:43.118168+0000","last_clean":"2023-10-03T18:32:43.118168+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:32:43.118168+0000","last_undegraded":"2023-10-03T18:32:43.118168+0000","last_fullsized":"2023-10-03T18:32:43.118168+0000","mapping_epoch":12,"log_start":"425'216290","ondisk_log_start":"425'216290","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:32:44.050089+0000","scrub_duration":370.574046178,"objects_trimmed":6,"snaptrim_duration":5.0000622420000003,"stat_sum":{"num_bytes":2924151606,"num_objects":2946,"num_object_clones":2303,"num_object_copies":5892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":200141,"num_read_kb":686375,"num_write":169250,"num_write_kb":672886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"3"}]},{"pgid":"2.4","version":"458'208750","reported_seq":361944,"reported_epoch":458,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:32:46.008553+0000","last_change":"2023-10-03T18:32:45.755994+0000","last_active":"2023-10-03T18:32:46.008553+0000","last_peered":"2023-10-03T18:32:46.008553+0000","last_clean":"2023-10-03T18:32:46.008553+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:32:46.008553+0000","last_undegraded":"2023-10-03T18:32:46.008553+0000","last_fullsized":"2023-10-03T18:32:46.008553+0000","mapping_epoch":12,"log_start":"414'198709","ondisk_log_start":"414'198709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":266,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":2.9673506380000001,"stat_sum":{"num_bytes":2911880008,"num_objects":10883,"num_object_clones":10243,"num_object_copies":21766,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10883,"num_whiteouts":0,"num_read":184383,"num_read_kb":668856,"num_write":167259,"num_write_kb":666720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ee","length":"3"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"4"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24176825617,"num_objects":74290,"num_object_clones":69026,"num_object_copies":148580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74290,"num_whiteouts":0,"num_read":1549794,"num_read_kb":5584391,"num_write":1391435,"num_write_kb":5543897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11342917632,"data_stored":139692390946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":78122,"internal_metadata":0},"log_size":80334,"ondisk_log_size":80334,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739088,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4186820,"kb_used_data":2131772,"kb_used_omap":1494,"kb_used_meta":2053545,"kb_avail":90185020,"statfs":{"total":96636764160,"available":92349460480,"internally_reserved":0,"allocated":2182934528,"data_stored":22107832013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1530832,"internal_metadata":2102830128},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,2,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.7729999999999999,"15min":2.2130000000000001},"min":{"1min":0.20499999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":7.6390000000000002,"5min":14.26,"15min":15.964},"last":0.311},{"interface":"front","average":{"1min":1.984,"5min":2.0139999999999998,"15min":2.3599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.7829999999999995,"5min":12.082000000000001,"15min":34.176000000000002},"last":14.27}]},{"osd":1,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":2.117,"15min":2.4380000000000002},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.8449999999999998,"5min":13.836,"15min":17.274000000000001},"last":15.114000000000001},{"interface":"front","average":{"1min":1.6619999999999999,"5min":2.0059999999999998,"15min":2.3140000000000001},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.505000000000001,"5min":13.462999999999999,"15min":13.842000000000001},"last":2.581}]}]},{"osd":0,"up_from":8,"seq":34359739086,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5218132,"kb_used_data":3415564,"kb_used_omap":2774,"kb_used_meta":1799785,"kb_avail":89153708,"statfs":{"total":96636764160,"available":91293396992,"internally_reserved":0,"allocated":3497537536,"data_stored":47745167034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1842980447},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,0,0,1,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":3.3290000000000002,"5min":3.4209999999999998,"15min":3.0630000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.14799999999999999},"max":{"1min":11.391,"5min":21.256,"15min":45.886000000000003},"last":4.7999999999999998},{"interface":"front","average":{"1min":2.9049999999999998,"5min":4.0919999999999996,"15min":3.4359999999999999},"min":{"1min":0.222,"5min":0.20399999999999999,"15min":0.20100000000000001},"max":{"1min":17.323,"5min":20.763000000000002,"15min":45.555},"last":4.8220000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:31:53 2023","interfaces":[{"interface":"back","average":{"1min":2.335,"5min":1.9990000000000001,"15min":1.946},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":8.1519999999999992,"5min":8.9860000000000007,"15min":13.058},"last":0.316},{"interface":"front","average":{"1min":3.0859999999999999,"5min":3.4009999999999998,"15min":2.7160000000000002},"min":{"1min":0.25700000000000001,"5min":0.215,"15min":0.192},"max":{"1min":14.611000000000001,"5min":25.77,"15min":25.77},"last":0.25600000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739054,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7299168,"kb_used_data":5543768,"kb_used_omap":4493,"kb_used_meta":1750898,"kb_avail":87072672,"statfs":{"total":96636764160,"available":89162416128,"internally_reserved":0,"allocated":5676818432,"data_stored":69854002612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1792919992},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,3,2,1,15,7,10],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":3.1230000000000002,"5min":4.5259999999999998,"15min":4.0410000000000004},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":10.069000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":2.339,"5min":3.202,"15min":3.1269999999999998},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":15.702999999999999,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.26600000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":2.1589999999999998,"15min":2.4649999999999999},"min":{"1min":0.24199999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.5830000000000002,"5min":14.436999999999999,"15min":34.395000000000003},"last":1.0600000000000001},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.3170000000000002,"15min":2.673},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.0869999999999997,"5min":14.926,"15min":36.332999999999998},"last":0.23200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3492110336,"data_stored":47726939335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5672632320,"data_stored":69871093009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2178174976,"data_stored":22094358602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10287,"internal_metadata":0}]}} 2023-10-03T18:32:47.601 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:47.601 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:47.601 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:47.602 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:32:47.602 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:32:48.255 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.253+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:48.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.333+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:48.335 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.333+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:48.354 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.352+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:48.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.389+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:48.391 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.389+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:48.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.396+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:48.398 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.396+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:48.445 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:48.443+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:49.313 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.309+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:49.349 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.347+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:49.352 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.350+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:49.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.362+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:49.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:49.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:49.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:49.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:49.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:49.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:49.366+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:50.544 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.536+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:50.562 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.556+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:50.562 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.556+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:50.645 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.643+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:50.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.667+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:50.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.693+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:50.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.693+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:50.697 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.693+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:50.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.933+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:50.936 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:50.933+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:53.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.914+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:53.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.929+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:53.931 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.929+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:53.932 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.929+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:53.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.992+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:53.995 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.992+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:53.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.994+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:53.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.994+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:53.996 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:53.994+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:57.051 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.050+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:57.167 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.162+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:57.340 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.338+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:57.360 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.353+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:57.631 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.627+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:32:57.683 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.682+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:32:57.726 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.722+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:32:57.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.757+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:57.766 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.761+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:57.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.768+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:57.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.768+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:57.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.768+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:57.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.768+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:57.770 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:57.768+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:32:59.199 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.198+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:32:59.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:32:59.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:32:59.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:32:59.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:32:59.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:32:59.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:32:59.245+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:00.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.474+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:00.479 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.475+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:00.479 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.475+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:00.531 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.527+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:00.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.527+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:00.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.527+0000 7fc515f98700 -1 reset not still connected to 0x56220ca65e10 2023-10-03T18:33:00.532 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.527+0000 7fc515f98700 -1 reset not still connected to 0x56220f5de0d0 2023-10-03T18:33:00.544 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.527+0000 7fc515f98700 -1 reset not still connected to 0x562210b55450 2023-10-03T18:33:00.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:00.590+0000 7fc515f98700 -1 reset not still connected to 0x562221bdedd0 2023-10-03T18:33:01.471 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:01.470+0000 7f9f21f20700 -1 reset not still connected to 0x5632dd528f70 2023-10-03T18:33:01.503 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.503+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:01.588 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.586+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:01.600 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.599+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:01.709 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:01.708+0000 7f9f21f20700 -1 reset not still connected to 0x5632d23e5860 2023-10-03T18:33:01.709 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:01.708+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2675860 2023-10-03T18:33:01.723 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.713+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:01.723 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:01.721+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:33:01.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.720+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:01.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.723+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:01.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.723+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:01.724 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.723+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:01.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.724+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:01.727 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:01.724+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:07.603 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:33:08.311 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:33:08.319 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:33:08.333 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2269,"stamp":"2023-10-03T18:33:07.811270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24201942934,"num_objects":74167,"num_object_clones":68899,"num_object_copies":148334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74167,"num_whiteouts":0,"num_read":1558556,"num_read_kb":5612347,"num_write":1397702,"num_write_kb":5569841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80414,"ondisk_log_size":80414,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":16684788,"kb_used_data":11142408,"kb_used_omap":8765,"kb_used_meta":5533570,"kb_avail":266430732,"statfs":{"total":289910292480,"available":272825069568,"internally_reserved":0,"allocated":11409825792,"data_stored":139775367461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8975886,"internal_metadata":5666376178},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,5,12,9,27,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1752989,"num_objects":87,"num_object_clones":89,"num_object_copies":174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":87,"num_whiteouts":0,"num_read":1378,"num_read_kb":2576,"num_write":509,"num_write_kb":1779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015145"},"pg_stats":[{"pgid":"2.7","version":"461'214667","reported_seq":373215,"reported_epoch":461,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:02.221773+0000","last_change":"2023-10-03T18:33:00.557395+0000","last_active":"2023-10-03T18:33:02.221773+0000","last_peered":"2023-10-03T18:33:02.221773+0000","last_clean":"2023-10-03T18:33:02.221773+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:33:02.221773+0000","last_undegraded":"2023-10-03T18:33:02.221773+0000","last_fullsized":"2023-10-03T18:33:02.221773+0000","mapping_epoch":12,"log_start":"418'204625","ondisk_log_start":"418'204625","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":349,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:53:11.959472+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":7.7489239130000005,"stat_sum":{"num_bytes":3014476210,"num_objects":13612,"num_object_clones":12962,"num_object_copies":27224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13612,"num_whiteouts":0,"num_read":189534,"num_read_kb":697695,"num_write":173574,"num_write_kb":692052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f0","length":"1"},{"start":"1f2","length":"1"}]},{"pgid":"2.6","version":"461'198467","reported_seq":346253,"reported_epoch":461,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:02.229187+0000","last_change":"2023-10-03T18:33:00.385086+0000","last_active":"2023-10-03T18:33:02.229187+0000","last_peered":"2023-10-03T18:33:02.229187+0000","last_clean":"2023-10-03T18:33:02.229187+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:33:02.229187+0000","last_undegraded":"2023-10-03T18:33:02.229187+0000","last_fullsized":"2023-10-03T18:33:02.229187+0000","mapping_epoch":12,"log_start":"415'188400","ondisk_log_start":"415'188400","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10067,"log_dups_size":0,"ondisk_log_size":10067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":263,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":7.5680359560000001,"stat_sum":{"num_bytes":2784973574,"num_objects":10124,"num_object_clones":9517,"num_object_copies":20248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10124,"num_whiteouts":0,"num_read":175844,"num_read_kb":644136,"num_write":160253,"num_write_kb":639476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f0","length":"1"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"4"}]},{"pgid":"2.5","version":"461'238121","reported_seq":412344,"reported_epoch":461,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:02.187371+0000","last_change":"2023-10-03T18:32:58.566357+0000","last_active":"2023-10-03T18:33:02.187371+0000","last_peered":"2023-10-03T18:33:02.187371+0000","last_clean":"2023-10-03T18:33:02.187371+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:33:02.187371+0000","last_undegraded":"2023-10-03T18:33:02.187371+0000","last_fullsized":"2023-10-03T18:33:02.187371+0000","mapping_epoch":12,"log_start":"426'228106","ondisk_log_start":"426'228106","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10015,"log_dups_size":0,"ondisk_log_size":10015,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":299,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":5.7462502789999998,"stat_sum":{"num_bytes":3267355076,"num_objects":13073,"num_object_clones":12367,"num_object_copies":26146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13073,"num_whiteouts":0,"num_read":217903,"num_read_kb":763317,"num_write":189186,"num_write_kb":752590,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f0","length":"1"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"8"}]},{"pgid":"2.0","version":"465'238067","reported_seq":401158,"reported_epoch":465,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:06.907418+0000","last_change":"2023-10-03T18:33:05.586663+0000","last_active":"2023-10-03T18:33:06.907418+0000","last_peered":"2023-10-03T18:33:06.907418+0000","last_clean":"2023-10-03T18:33:06.907418+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:33:06.907418+0000","last_undegraded":"2023-10-03T18:33:06.907418+0000","last_fullsized":"2023-10-03T18:33:06.907418+0000","mapping_epoch":12,"log_start":"431'228056","ondisk_log_start":"431'228056","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":163,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T23:32:55.575158+0000","scrub_duration":734.24538164700004,"objects_trimmed":12,"snaptrim_duration":0.62555309299999995,"stat_sum":{"num_bytes":3157148253,"num_objects":7946,"num_object_clones":7262,"num_object_copies":15892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7946,"num_whiteouts":0,"num_read":197446,"num_read_kb":727965,"num_write":182601,"num_write_kb":728073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":628,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-10-03T18:33:01.019028+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:33:01.019028+0000","last_peered":"2023-10-03T18:33:01.019028+0000","last_clean":"2023-10-03T18:33:01.019028+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:33:01.019028+0000","last_undegraded":"2023-10-03T18:33:01.019028+0000","last_fullsized":"2023-10-03T18:33:01.019028+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"461'229035","reported_seq":395913,"reported_epoch":461,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:02.106714+0000","last_change":"2023-10-03T18:33:00.563747+0000","last_active":"2023-10-03T18:33:02.106714+0000","last_peered":"2023-10-03T18:33:02.106714+0000","last_clean":"2023-10-03T18:33:02.106714+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:33:02.106714+0000","last_undegraded":"2023-10-03T18:33:02.106714+0000","last_fullsized":"2023-10-03T18:33:02.106714+0000","mapping_epoch":12,"log_start":"425'219024","ondisk_log_start":"425'219024","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10011,"log_dups_size":0,"ondisk_log_size":10011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":289,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":5.6205455820000001,"stat_sum":{"num_bytes":3209360301,"num_objects":12523,"num_object_clones":11822,"num_object_copies":25046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12523,"num_whiteouts":0,"num_read":200615,"num_read_kb":739617,"num_write":184769,"num_write_kb":736747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f0","length":"1"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"1"}]},{"pgid":"2.1","version":"465'227665","reported_seq":378752,"reported_epoch":465,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:06.876151+0000","last_change":"2023-10-03T18:33:05.983087+0000","last_active":"2023-10-03T18:33:06.876151+0000","last_peered":"2023-10-03T18:33:06.876151+0000","last_clean":"2023-10-03T18:33:06.876151+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:33:06.876151+0000","last_undegraded":"2023-10-03T18:33:06.876151+0000","last_fullsized":"2023-10-03T18:33:06.876151+0000","mapping_epoch":12,"log_start":"427'217646","ondisk_log_start":"427'217646","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10019,"log_dups_size":0,"ondisk_log_size":10019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:01:13.369619+0000","scrub_duration":424.08306342600002,"objects_trimmed":12,"snaptrim_duration":0.019930798,"stat_sum":{"num_bytes":2925752942,"num_objects":3051,"num_object_clones":2416,"num_object_copies":6102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":190206,"num_read_kb":677039,"num_write":169099,"num_write_kb":673267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"2"}]},{"pgid":"2.2","version":"464'227458","reported_seq":384722,"reported_epoch":464,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:05.421849+0000","last_change":"2023-10-03T18:33:04.491787+0000","last_active":"2023-10-03T18:33:05.421849+0000","last_peered":"2023-10-03T18:33:05.421849+0000","last_clean":"2023-10-03T18:33:05.421849+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:33:05.421849+0000","last_undegraded":"2023-10-03T18:33:05.421849+0000","last_fullsized":"2023-10-03T18:33:05.421849+0000","mapping_epoch":12,"log_start":"427'217393","ondisk_log_start":"427'217393","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:18:07.298975+0000","scrub_duration":370.574046178,"objects_trimmed":10,"snaptrim_duration":0.66479500400000002,"stat_sum":{"num_bytes":2927616876,"num_objects":2998,"num_object_clones":2354,"num_object_copies":5996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":201715,"num_read_kb":690728,"num_write":170138,"num_write_kb":676298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"1"}]},{"pgid":"2.4","version":"461'209696","reported_seq":363570,"reported_epoch":461,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:02.100818+0000","last_change":"2023-10-03T18:33:00.406201+0000","last_active":"2023-10-03T18:33:02.100818+0000","last_peered":"2023-10-03T18:33:02.100818+0000","last_clean":"2023-10-03T18:33:02.100818+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:33:02.100818+0000","last_undegraded":"2023-10-03T18:33:02.100818+0000","last_fullsized":"2023-10-03T18:33:02.100818+0000","mapping_epoch":12,"log_start":"417'199611","ondisk_log_start":"417'199611","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":271,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":3.5081148529999999,"stat_sum":{"num_bytes":2914669334,"num_objects":10838,"num_object_clones":10199,"num_object_copies":21676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10838,"num_whiteouts":0,"num_read":185187,"num_read_kb":671759,"num_write":167945,"num_write_kb":669444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1f0","length":"1"},{"start":"1f2","length":"1"},{"start":"1f5","length":"2"},{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24201352566,"num_objects":74165,"num_object_clones":68899,"num_object_copies":148330,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74165,"num_whiteouts":0,"num_read":1558450,"num_read_kb":5612256,"num_write":1397565,"num_write_kb":5567947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11394183168,"data_stored":139925514988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80315,"ondisk_log_size":80315,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739092,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3994396,"kb_used_data":2141836,"kb_used_omap":1497,"kb_used_meta":1851046,"kb_avail":90377444,"statfs":{"total":96636764160,"available":92546502656,"internally_reserved":0,"allocated":2193240064,"data_stored":22115372805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1533477,"internal_metadata":1895471579},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1,1,0,5,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.7729999999999999,"15min":2.2130000000000001},"min":{"1min":0.20499999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":7.6390000000000002,"5min":14.26,"15min":15.964},"last":0.28999999999999998},{"interface":"front","average":{"1min":1.984,"5min":2.0139999999999998,"15min":2.3599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.7829999999999995,"5min":12.082000000000001,"15min":34.176000000000002},"last":1.1240000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":2.117,"15min":2.4380000000000002},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.8449999999999998,"5min":13.836,"15min":17.274000000000001},"last":1.171},{"interface":"front","average":{"1min":1.6619999999999999,"5min":2.0059999999999998,"15min":2.3140000000000001},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.505000000000001,"5min":13.462999999999999,"15min":13.842000000000001},"last":1.077}]}]},{"osd":0,"up_from":8,"seq":34359739091,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5304092,"kb_used_data":3433556,"kb_used_omap":2774,"kb_used_meta":1867753,"kb_avail":89067748,"statfs":{"total":96636764160,"available":91205373952,"internally_reserved":0,"allocated":3515961344,"data_stored":47924235562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":1912579679},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":3.3799999999999999,"5min":3.1179999999999999,"15min":3.0920000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":10.433999999999999,"5min":19.873000000000001,"15min":45.886000000000003},"last":1.1910000000000001},{"interface":"front","average":{"1min":3.6880000000000002,"5min":4.0599999999999996,"15min":3.524},"min":{"1min":0.22,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":13.065,"5min":20.763000000000002,"15min":45.555},"last":1.4750000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":1.996,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":7.9080000000000004,"5min":8.1519999999999992,"15min":13.058},"last":0.28699999999999998},{"interface":"front","average":{"1min":2.6080000000000001,"5min":2.8380000000000001,"15min":2.7530000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.192},"max":{"1min":9.7430000000000003,"5min":18.352,"15min":25.77},"last":0.46500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359739057,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7386300,"kb_used_data":5567016,"kb_used_omap":4493,"kb_used_meta":1814770,"kb_avail":86985540,"statfs":{"total":96636764160,"available":89073192960,"internally_reserved":0,"allocated":5700624384,"data_stored":69735759094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1858324920},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,2,4,11,8,22],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":3.1230000000000002,"5min":4.5259999999999998,"15min":4.0410000000000004},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":10.069000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":2.0430000000000001},{"interface":"front","average":{"1min":2.339,"5min":3.202,"15min":3.1269999999999998},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":15.702999999999999,"5min":52.982999999999997,"15min":52.982999999999997},"last":2.653}]},{"osd":2,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":2.1589999999999998,"15min":2.4649999999999999},"min":{"1min":0.24199999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.5830000000000002,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.27100000000000002},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.3170000000000002,"15min":2.673},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.0869999999999997,"5min":14.926,"15min":36.332999999999998},"last":2.516}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3510505472,"data_stored":47984262399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5695385600,"data_stored":69779449205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2188292096,"data_stored":22161803384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:33:08.335 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:08.335 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:08.335 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:08.336 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:08.336 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.6 2023-10-03T18:33:08.983 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.6 on osd.1 to deep-scrub 2023-10-03T18:33:09.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.5 2023-10-03T18:33:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.5 on osd.1 to deep-scrub 2023-10-03T18:33:09.666 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.3 2023-10-03T18:33:10.318 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.3 on osd.1 to deep-scrub 2023-10-03T18:33:10.338 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.4 2023-10-03T18:33:11.010 INFO:teuthology.orchestra.run.smithi031.stderr:instructing pg 2.4 on osd.1 to deep-scrub 2023-10-03T18:33:11.023 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:33:11.958 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:11.957+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:12.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.005+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:12.006 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.005+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:12.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.079+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:12.660 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.660+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:12.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.660+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:12.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.660+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:12.661 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.660+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:12.746 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.743+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:12.757 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:12.753+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:13.040 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.039+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:13.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.083+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:13.083 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.083+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:13.104 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.098+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:13.115 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.114+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:13.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.195+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:13.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.195+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:33:13.196 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.195+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:13.371 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.363+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:13.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.420+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:13.421 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.420+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:13.530 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.529+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:13.530 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.529+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:13.530 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.529+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:13.636 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.634+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:13.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:13.639+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:16.400 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.398+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:16.461 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.455+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:16.569 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.568+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:16.696 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.694+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:16.863 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.861+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:16.864 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:16.862+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:17.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:17.117+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:17.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:17.117+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:17.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:17.117+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:18.230 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.229+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:18.281 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.280+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:18.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.355+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:18.357 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.355+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:18.370 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:18.368+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:33:18.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.481+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:18.544 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.539+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:18.545 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.539+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:18.610 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.610+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:18.637 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:18.636+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:19.494 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.493+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:33:19.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.522+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:19.524 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.522+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:19.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.596+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:19.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.704+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:19.711 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.704+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:19.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.754+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:19.755 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.754+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:19.862 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.858+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:20.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.999+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:20.001 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:19.999+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:21.909 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:21.907+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:22.027 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:22.021+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:22.091 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:22.090+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:27.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.249+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:27.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.249+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:27.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.249+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:27.322 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.319+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:27.835 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.833+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:27.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.987+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:27.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:27.987+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:28.033 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.029+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:28.216 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.215+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:28.240 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.239+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:28.299 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.297+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:28.324 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.324+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:28.325 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.324+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:28.460 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.459+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:28.488 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.479+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:33:28.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.495+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:28.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.495+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:28.496 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.495+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:28.547 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.543+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:28.548 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.546+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:28.594 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.591+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:28.595 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.591+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:28.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.596+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:28.597 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.596+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:28.643 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.643+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:28.829 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.822+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:28.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.852+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:28.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.852+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:28.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.853+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:28.917 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.912+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:28.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.917+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:28.934 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.932+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:28.953 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.949+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:28.966 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:28.965+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:29.326 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.324+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:29.327 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.324+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:29.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.344+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:29.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.344+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:29.444 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.443+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:29.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.539+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:29.540 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.539+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:29.569 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:29.564+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:33:29.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.658+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:29.659 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.658+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:29.682 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:29.682+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:30.052 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:33:30.051+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:33:31.023 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:33:32.005 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:33:32.020 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:33:32.029 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2284,"stamp":"2023-10-03T18:33:31.548579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24224896817,"num_objects":75952,"num_object_clones":70686,"num_object_copies":151904,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75952,"num_whiteouts":0,"num_read":1566832,"num_read_kb":5637663,"num_write":1403383,"num_write_kb":5592315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80430,"ondisk_log_size":80430,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17047368,"kb_used_data":11186920,"kb_used_omap":8765,"kb_used_meta":5851650,"kb_avail":266068152,"statfs":{"total":289910292480,"available":272453787648,"internally_reserved":0,"allocated":11455406080,"data_stored":144032939941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8975886,"internal_metadata":5992090098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,4,5,19,11,17,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9072640,"num_objects":385,"num_object_clones":385,"num_object_copies":770,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":385,"num_whiteouts":0,"num_read":2298,"num_read_kb":9164,"num_write":2211,"num_write_kb":8844,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.745870"},"pg_stats":[{"pgid":"2.7","version":"470'216174","reported_seq":375120,"reported_epoch":470,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:30.091081+0000","last_change":"2023-10-03T18:33:27.147292+0000","last_active":"2023-10-03T18:33:30.091081+0000","last_peered":"2023-10-03T18:33:30.091081+0000","last_clean":"2023-10-03T18:33:30.091081+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:33:30.091081+0000","last_undegraded":"2023-10-03T18:33:30.091081+0000","last_fullsized":"2023-10-03T18:33:30.091081+0000","mapping_epoch":12,"log_start":"427'206130","ondisk_log_start":"427'206130","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":359,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:31:59.559711+0000","scrub_duration":505.38444223300002,"objects_trimmed":28,"snaptrim_duration":0.92728864099999997,"stat_sum":{"num_bytes":3017744818,"num_objects":13936,"num_object_clones":13286,"num_object_copies":27872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13936,"num_whiteouts":0,"num_read":190614,"num_read_kb":701108,"num_write":174384,"num_write_kb":695252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"470'199806","reported_seq":348025,"reported_epoch":470,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:29.915329+0000","last_change":"2023-10-03T18:33:27.147299+0000","last_active":"2023-10-03T18:33:29.915329+0000","last_peered":"2023-10-03T18:33:29.915329+0000","last_clean":"2023-10-03T18:33:29.915329+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:33:29.915329+0000","last_undegraded":"2023-10-03T18:33:29.915329+0000","last_fullsized":"2023-10-03T18:33:29.915329+0000","mapping_epoch":12,"log_start":"424'189709","ondisk_log_start":"424'189709","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":272,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.301809596,"stat_sum":{"num_bytes":2787857108,"num_objects":10420,"num_object_clones":9814,"num_object_copies":20840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10420,"num_whiteouts":0,"num_read":176788,"num_read_kb":647300,"num_write":160960,"num_write_kb":642286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ff","length":"9"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"8"}]},{"pgid":"2.5","version":"470'239724","reported_seq":414421,"reported_epoch":470,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:29.975350+0000","last_change":"2023-10-03T18:33:28.115421+0000","last_active":"2023-10-03T18:33:29.975350+0000","last_peered":"2023-10-03T18:33:29.975350+0000","last_clean":"2023-10-03T18:33:29.975350+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:33:29.975350+0000","last_undegraded":"2023-10-03T18:33:29.975350+0000","last_fullsized":"2023-10-03T18:33:29.975350+0000","mapping_epoch":12,"log_start":"430'229712","ondisk_log_start":"430'229712","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":309,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":20,"snaptrim_duration":1.1093779880000001,"stat_sum":{"num_bytes":3270726135,"num_objects":13429,"num_object_clones":12722,"num_object_copies":26858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13429,"num_whiteouts":0,"num_read":219297,"num_read_kb":767111,"num_write":190026,"num_write_kb":755889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ff","length":"9"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"1"}]},{"pgid":"2.0","version":"469'239472","reported_seq":402789,"reported_epoch":469,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:27.028845+0000","last_change":"2023-10-03T18:33:26.182631+0000","last_active":"2023-10-03T18:33:27.028845+0000","last_peered":"2023-10-03T18:33:27.028845+0000","last_clean":"2023-10-03T18:33:27.028845+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:33:27.028845+0000","last_undegraded":"2023-10-03T18:33:27.028845+0000","last_fullsized":"2023-10-03T18:33:27.028845+0000","mapping_epoch":12,"log_start":"440'229466","ondisk_log_start":"440'229466","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10006,"log_dups_size":0,"ondisk_log_size":10006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":155,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:30:15.029430+0000","scrub_duration":734.24538164700004,"objects_trimmed":65,"snaptrim_duration":3.3513660380000001,"stat_sum":{"num_bytes":3159462493,"num_objects":8009,"num_object_clones":7325,"num_object_copies":16018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8009,"num_whiteouts":0,"num_read":198184,"num_read_kb":730485,"num_write":183165,"num_write_kb":730329,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":637,"reported_epoch":469,"state":"active+clean","last_fresh":"2023-10-03T18:33:22.845801+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:33:22.845801+0000","last_peered":"2023-10-03T18:33:22.845801+0000","last_clean":"2023-10-03T18:33:22.845801+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:33:22.845801+0000","last_undegraded":"2023-10-03T18:33:22.845801+0000","last_fullsized":"2023-10-03T18:33:22.845801+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"470'230556","reported_seq":397855,"reported_epoch":470,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:29.963692+0000","last_change":"2023-10-03T18:33:28.043312+0000","last_active":"2023-10-03T18:33:29.963692+0000","last_peered":"2023-10-03T18:33:29.963692+0000","last_clean":"2023-10-03T18:33:29.963692+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:33:29.963692+0000","last_undegraded":"2023-10-03T18:33:29.963692+0000","last_fullsized":"2023-10-03T18:33:29.963692+0000","mapping_epoch":12,"log_start":"428'220533","ondisk_log_start":"428'220533","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10023,"log_dups_size":0,"ondisk_log_size":10023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":298,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.12981137400000001,"stat_sum":{"num_bytes":3212616621,"num_objects":12856,"num_object_clones":12155,"num_object_copies":25712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12856,"num_whiteouts":0,"num_read":201769,"num_read_kb":743135,"num_write":185583,"num_write_kb":739947,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ff","length":"9"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"2"}]},{"pgid":"2.1","version":"469'229029","reported_seq":380329,"reported_epoch":469,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:27.064297+0000","last_change":"2023-10-03T18:33:25.168770+0000","last_active":"2023-10-03T18:33:27.064297+0000","last_peered":"2023-10-03T18:33:27.064297+0000","last_clean":"2023-10-03T18:33:27.064297+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:33:27.064297+0000","last_undegraded":"2023-10-03T18:33:27.064297+0000","last_fullsized":"2023-10-03T18:33:27.064297+0000","mapping_epoch":12,"log_start":"431'218957","ondisk_log_start":"431'218957","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10072,"log_dups_size":0,"ondisk_log_size":10072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:19:49.162060+0000","scrub_duration":424.08306342600002,"objects_trimmed":71,"snaptrim_duration":2.3305624919999999,"stat_sum":{"num_bytes":2927780462,"num_objects":3066,"num_object_clones":2431,"num_object_copies":6132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3066,"num_whiteouts":0,"num_read":190960,"num_read_kb":679387,"num_write":169600,"num_write_kb":675260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1fe","length":"a"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"}]},{"pgid":"2.2","version":"470'229216","reported_seq":386822,"reported_epoch":470,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:30.705610+0000","last_change":"2023-10-03T18:33:30.007500+0000","last_active":"2023-10-03T18:33:30.705610+0000","last_peered":"2023-10-03T18:33:30.705610+0000","last_clean":"2023-10-03T18:33:30.705610+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:33:30.705610+0000","last_undegraded":"2023-10-03T18:33:30.705610+0000","last_fullsized":"2023-10-03T18:33:30.705610+0000","mapping_epoch":12,"log_start":"431'219204","ondisk_log_start":"431'219204","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:25.885705+0000","scrub_duration":370.574046178,"objects_trimmed":50,"snaptrim_duration":2.1681630090000001,"stat_sum":{"num_bytes":2930488172,"num_objects":3060,"num_object_clones":2416,"num_object_copies":6120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3060,"num_whiteouts":0,"num_read":202788,"num_read_kb":693978,"num_write":170841,"num_write_kb":679110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1ff","length":"9"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"1"}]},{"pgid":"2.4","version":"470'211081","reported_seq":365384,"reported_epoch":470,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:29.968349+0000","last_change":"2023-10-03T18:33:25.543567+0000","last_active":"2023-10-03T18:33:29.968349+0000","last_peered":"2023-10-03T18:33:29.968349+0000","last_clean":"2023-10-03T18:33:29.968349+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:33:29.968349+0000","last_undegraded":"2023-10-03T18:33:29.968349+0000","last_fullsized":"2023-10-03T18:33:29.968349+0000","mapping_epoch":12,"log_start":"426'201016","ondisk_log_start":"426'201016","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10065,"log_dups_size":0,"ondisk_log_size":10065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":280,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.6981485520000001,"stat_sum":{"num_bytes":2917630640,"num_objects":11174,"num_object_clones":10537,"num_object_copies":22348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11174,"num_whiteouts":0,"num_read":186326,"num_read_kb":675068,"num_write":168687,"num_write_kb":672348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1ff","length":"9"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24224306449,"num_objects":75950,"num_object_clones":70686,"num_object_copies":151900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75950,"num_whiteouts":0,"num_read":1566726,"num_read_kb":5637572,"num_write":1403246,"num_write_kb":5590421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11440365568,"data_stored":144074144290,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80331,"ondisk_log_size":80331,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739096,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4095372,"kb_used_data":2148420,"kb_used_omap":1497,"kb_used_meta":1945446,"kb_avail":90276468,"statfs":{"total":96636764160,"available":92443103232,"internally_reserved":0,"allocated":2199982080,"data_stored":22589023321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1533477,"internal_metadata":1992137179},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,0,2,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.7729999999999999,"15min":2.2130000000000001},"min":{"1min":0.20499999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":7.6390000000000002,"5min":14.26,"15min":15.964},"last":2.734},{"interface":"front","average":{"1min":1.984,"5min":2.0139999999999998,"15min":2.3599999999999999},"min":{"1min":0.22600000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":9.7829999999999995,"5min":12.082000000000001,"15min":34.176000000000002},"last":0.35299999999999998}]},{"osd":1,"last update":"Tue Oct 3 18:32:32 2023","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":2.117,"15min":2.4380000000000002},"min":{"1min":0.28899999999999998,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":7.8449999999999998,"5min":13.836,"15min":17.274000000000001},"last":20.600999999999999},{"interface":"front","average":{"1min":1.6619999999999999,"5min":2.0059999999999998,"15min":2.3140000000000001},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":10.505000000000001,"5min":13.462999999999999,"15min":13.842000000000001},"last":11.028}]}]},{"osd":0,"up_from":8,"seq":34359739096,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5480824,"kb_used_data":3447344,"kb_used_omap":2774,"kb_used_meta":2030697,"kb_avail":88891016,"statfs":{"total":96636764160,"available":91024400384,"internally_reserved":0,"allocated":3530080256,"data_stored":49430454634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":2079434335},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2,8,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":3.3799999999999999,"5min":3.1179999999999999,"15min":3.0920000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":10.433999999999999,"5min":19.873000000000001,"15min":45.886000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":3.6880000000000002,"5min":4.0599999999999996,"15min":3.524},"min":{"1min":0.22,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":13.065,"5min":20.763000000000002,"15min":45.555},"last":0.84299999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":1.996,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":7.9080000000000004,"5min":8.1519999999999992,"15min":13.058},"last":2.6589999999999998},{"interface":"front","average":{"1min":2.6080000000000001,"5min":2.8380000000000001,"15min":2.7530000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.192},"max":{"1min":9.7430000000000003,"5min":18.352,"15min":25.77},"last":1.1950000000000001}]}]},{"osd":1,"up_from":8,"seq":34359739062,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7471172,"kb_used_data":5591156,"kb_used_omap":4493,"kb_used_meta":1875506,"kb_avail":86900668,"statfs":{"total":96636764160,"available":88986284032,"internally_reserved":0,"allocated":5725343744,"data_stored":72013461986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":1920518584},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,3,11,7,13,6,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":3.1230000000000002,"5min":4.5259999999999998,"15min":4.0410000000000004},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":10.069000000000001,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":2.339,"5min":3.202,"15min":3.1269999999999998},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.19900000000000001},"max":{"1min":15.702999999999999,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:32:42 2023","interfaces":[{"interface":"back","average":{"1min":1.625,"5min":2.1589999999999998,"15min":2.4649999999999999},"min":{"1min":0.24199999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":6.5830000000000002,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.316},{"interface":"front","average":{"1min":2.2469999999999999,"5min":2.3170000000000002,"15min":2.673},"min":{"1min":0.28499999999999998,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":9.0869999999999997,"5min":14.926,"15min":36.332999999999998},"last":0.35699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3524800512,"data_stored":49450650777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5720547328,"data_stored":72039130385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2195017728,"data_stored":22584363128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:33:32.031 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:32.031 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:32.031 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:32.032 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:32.032 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:33:34.759 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.758+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:34.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.758+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:34.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.758+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:34.760 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.758+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:34.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.803+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:34.806 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:34.803+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:35.617 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.614+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:35.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.687+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:35.705 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.687+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:35.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.694+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:35.706 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.694+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:35.779 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.777+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:35.804 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.802+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:35.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:35.802+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:36.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.370+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:36.372 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.370+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:36.412 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.408+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:36.413 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.411+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:36.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.526+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:36.529 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.526+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:36.762 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:36.761+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:39.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:39.985+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:39.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:39.985+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:39.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:39.985+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:39.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:39.985+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:39.988 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:39.985+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:40.005 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.003+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:40.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.004+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:40.013 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.004+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:40.374 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.372+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:40.387 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.384+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:40.387 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.384+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:40.407 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.399+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:40.419 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.414+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:40.450 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:40.448+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:41.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.099+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:41.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.099+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:41.282 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.280+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:41.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.304+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:41.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.304+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:41.308 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.304+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:41.309 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.304+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:41.499 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.497+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:41.514 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:41.512+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:44.955 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:44.953+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:45.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:45.045+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:45.049 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:45.045+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:47.805 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:47.799+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:47.834 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:47.832+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:47.834 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:47.832+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:47.834 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:47.832+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:47.860 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:47.859+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:49.008 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.006+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:49.044 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.042+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:49.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.098+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:49.101 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.100+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:49.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.123+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:49.125 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.123+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:49.159 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.153+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:49.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.160+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:49.162 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.160+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:49.906 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.905+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:49.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.912+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:49.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.912+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:49.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.914+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:49.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.918+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:49.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.918+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:49.933 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.931+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:49.940 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.935+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:49.969 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:49.966+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:51.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.470+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:33:51.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.470+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:51.475 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.472+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:51.518 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.516+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:51.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.552+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:51.558 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.552+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:51.561 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.560+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:51.572 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.568+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:51.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.600+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:33:51.605 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:51.600+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:52.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:33:52.800 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:33:52.813 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:33:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2296,"stamp":"2023-10-03T18:33:51.567577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24249902897,"num_objects":75858,"num_object_clones":70592,"num_object_copies":151716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75858,"num_whiteouts":0,"num_read":1573047,"num_read_kb":5662443,"num_write":1409466,"num_write_kb":5616647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80598,"ondisk_log_size":80598,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17317824,"kb_used_data":11235048,"kb_used_omap":8765,"kb_used_meta":6073986,"kb_avail":265797696,"statfs":{"total":289910292480,"available":272176840704,"internally_reserved":0,"allocated":11504689152,"data_stored":144201125717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8975886,"internal_metadata":6219762162},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4,6,12,3,18,9],"upper_bound":256},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13484032,"num_objects":-160,"num_object_clones":-160,"num_object_copies":-320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-160,"num_whiteouts":0,"num_read":3348,"num_read_kb":13336,"num_write":3282,"num_write_kb":13128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.010844"},"pg_stats":[{"pgid":"2.7","version":"472'217316","reported_seq":376878,"reported_epoch":472,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:51.192231+0000","last_change":"2023-10-03T18:33:48.409404+0000","last_active":"2023-10-03T18:33:51.192231+0000","last_peered":"2023-10-03T18:33:51.192231+0000","last_clean":"2023-10-03T18:33:51.192231+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:33:51.192231+0000","last_undegraded":"2023-10-03T18:33:51.192231+0000","last_fullsized":"2023-10-03T18:33:51.192231+0000","mapping_epoch":12,"log_start":"431'207233","ondisk_log_start":"431'207233","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10083,"log_dups_size":0,"ondisk_log_size":10083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":355,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T03:31:59.559711+0000","scrub_duration":505.38444223300002,"objects_trimmed":45,"snaptrim_duration":5.5429639650000002,"stat_sum":{"num_bytes":3020919218,"num_objects":13965,"num_object_clones":13315,"num_object_copies":27930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13965,"num_whiteouts":0,"num_read":191389,"num_read_kb":704204,"num_write":175151,"num_write_kb":698320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"201","length":"2"}]},{"pgid":"2.6","version":"472'200893","reported_seq":349686,"reported_epoch":472,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:51.157003+0000","last_change":"2023-10-03T18:33:50.001896+0000","last_active":"2023-10-03T18:33:51.157003+0000","last_peered":"2023-10-03T18:33:51.157003+0000","last_clean":"2023-10-03T18:33:51.157003+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:33:51.157003+0000","last_undegraded":"2023-10-03T18:33:51.157003+0000","last_fullsized":"2023-10-03T18:33:51.157003+0000","mapping_epoch":12,"log_start":"427'190814","ondisk_log_start":"427'190814","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10079,"log_dups_size":0,"ondisk_log_size":10079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":268,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":54,"snaptrim_duration":7.1656080659999999,"stat_sum":{"num_bytes":2790888148,"num_objects":10446,"num_object_clones":9840,"num_object_copies":20892,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10446,"num_whiteouts":0,"num_read":177520,"num_read_kb":650216,"num_write":161697,"num_write_kb":645234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"1"}]},{"pgid":"2.5","version":"472'240974","reported_seq":416369,"reported_epoch":472,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:51.184326+0000","last_change":"2023-10-03T18:33:50.001992+0000","last_active":"2023-10-03T18:33:51.184326+0000","last_peered":"2023-10-03T18:33:51.184326+0000","last_clean":"2023-10-03T18:33:51.184326+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:33:51.184326+0000","last_undegraded":"2023-10-03T18:33:51.184326+0000","last_fullsized":"2023-10-03T18:33:51.184326+0000","mapping_epoch":12,"log_start":"433'230915","ondisk_log_start":"433'230915","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10059,"log_dups_size":0,"ondisk_log_size":10059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":306,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":46,"snaptrim_duration":4.0241222810000004,"stat_sum":{"num_bytes":3274224119,"num_objects":13484,"num_object_clones":12777,"num_object_copies":26968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13484,"num_whiteouts":0,"num_read":220170,"num_read_kb":770583,"num_write":190879,"num_write_kb":759301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"2"}]},{"pgid":"2.0","version":"472'240944","reported_seq":404718,"reported_epoch":472,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:47.195388+0000","last_change":"2023-10-03T18:33:45.978005+0000","last_active":"2023-10-03T18:33:47.195388+0000","last_peered":"2023-10-03T18:33:47.195388+0000","last_clean":"2023-10-03T18:33:47.195388+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:33:47.195388+0000","last_undegraded":"2023-10-03T18:33:47.195388+0000","last_fullsized":"2023-10-03T18:33:47.195388+0000","mapping_epoch":12,"log_start":"442'230873","ondisk_log_start":"442'230873","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10071,"log_dups_size":0,"ondisk_log_size":10071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":147,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T22:30:15.029430+0000","scrub_duration":734.24538164700004,"objects_trimmed":72,"snaptrim_duration":3.1633041749999999,"stat_sum":{"num_bytes":3162427997,"num_objects":7922,"num_object_clones":7238,"num_object_copies":15844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7922,"num_whiteouts":0,"num_read":198919,"num_read_kb":733417,"num_write":183887,"num_write_kb":733217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"3"}]},{"pgid":"1.0","version":"88'99","reported_seq":639,"reported_epoch":471,"state":"active+clean","last_fresh":"2023-10-03T18:33:37.938757+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:33:37.938757+0000","last_peered":"2023-10-03T18:33:37.938757+0000","last_clean":"2023-10-03T18:33:37.938757+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:33:37.938757+0000","last_undegraded":"2023-10-03T18:33:37.938757+0000","last_fullsized":"2023-10-03T18:33:37.938757+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"472'231797","reported_seq":399772,"reported_epoch":472,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:33:51.117631+0000","last_change":"2023-10-03T18:33:48.659383+0000","last_active":"2023-10-03T18:33:51.117631+0000","last_peered":"2023-10-03T18:33:51.117631+0000","last_clean":"2023-10-03T18:33:51.117631+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:33:51.117631+0000","last_undegraded":"2023-10-03T18:33:51.117631+0000","last_fullsized":"2023-10-03T18:33:51.117631+0000","mapping_epoch":12,"log_start":"431'221736","ondisk_log_start":"431'221736","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10061,"log_dups_size":0,"ondisk_log_size":10061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":295,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":30,"snaptrim_duration":5.8232786650000001,"stat_sum":{"num_bytes":3216192429,"num_objects":12925,"num_object_clones":12224,"num_object_copies":25850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12925,"num_whiteouts":0,"num_read":202631,"num_read_kb":746563,"num_write":186452,"num_write_kb":743423,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"5"}]},{"pgid":"2.1","version":"472'230390","reported_seq":382176,"reported_epoch":472,"state":"active+clean","last_fresh":"2023-10-03T18:33:47.168916+0000","last_change":"2023-10-03T18:33:43.661122+0000","last_active":"2023-10-03T18:33:47.168916+0000","last_peered":"2023-10-03T18:33:47.168916+0000","last_clean":"2023-10-03T18:33:47.168916+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:33:47.168916+0000","last_undegraded":"2023-10-03T18:33:47.168916+0000","last_fullsized":"2023-10-03T18:33:47.168916+0000","mapping_epoch":12,"log_start":"440'220366","ondisk_log_start":"440'220366","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10024,"log_dups_size":0,"ondisk_log_size":10024,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T05:19:49.162060+0000","scrub_duration":424.08306342600002,"objects_trimmed":15,"snaptrim_duration":0.84769909099999996,"stat_sum":{"num_bytes":2930545262,"num_objects":2956,"num_object_clones":2321,"num_object_copies":5912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2956,"num_whiteouts":0,"num_read":191665,"num_read_kb":682203,"num_write":170273,"num_write_kb":677952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"}]},{"pgid":"2.2","version":"472'230488","reported_seq":388650,"reported_epoch":472,"state":"active+clean","last_fresh":"2023-10-03T18:33:50.838058+0000","last_change":"2023-10-03T18:33:44.396227+0000","last_active":"2023-10-03T18:33:50.838058+0000","last_peered":"2023-10-03T18:33:50.838058+0000","last_clean":"2023-10-03T18:33:50.838058+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:33:50.838058+0000","last_undegraded":"2023-10-03T18:33:50.838058+0000","last_fullsized":"2023-10-03T18:33:50.838058+0000","mapping_epoch":12,"log_start":"440'220410","ondisk_log_start":"440'220410","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10078,"log_dups_size":0,"ondisk_log_size":10078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:59:25.885705+0000","scrub_duration":370.574046178,"objects_trimmed":27,"snaptrim_duration":1.5552465170000001,"stat_sum":{"num_bytes":2933334892,"num_objects":2981,"num_object_clones":2337,"num_object_copies":5962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2981,"num_whiteouts":0,"num_read":203541,"num_read_kb":696990,"num_write":171536,"num_write_kb":681890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"}]},{"pgid":"2.4","version":"472'212265","reported_seq":367175,"reported_epoch":472,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:33:51.172968+0000","last_change":"2023-10-03T18:33:48.409511+0000","last_active":"2023-10-03T18:33:51.172968+0000","last_peered":"2023-10-03T18:33:51.172968+0000","last_clean":"2023-10-03T18:33:51.172968+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:33:51.172968+0000","last_undegraded":"2023-10-03T18:33:51.172968+0000","last_fullsized":"2023-10-03T18:33:51.172968+0000","mapping_epoch":12,"log_start":"430'202221","ondisk_log_start":"430'202221","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10044,"log_dups_size":0,"ondisk_log_size":10044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":277,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":54,"snaptrim_duration":1.4569968200000001,"stat_sum":{"num_bytes":2920780464,"num_objects":11177,"num_object_clones":10540,"num_object_copies":22354,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11177,"num_whiteouts":0,"num_read":187106,"num_read_kb":678176,"num_write":169454,"num_write_kb":675416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"201","length":"7"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"3"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24249312529,"num_objects":75856,"num_object_clones":70592,"num_object_copies":151712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75856,"num_whiteouts":0,"num_read":1572941,"num_read_kb":5662352,"num_write":1409329,"num_write_kb":5614753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11489849344,"data_stored":144196217378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80499,"ondisk_log_size":80499,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739100,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4117752,"kb_used_data":2156912,"kb_used_omap":1497,"kb_used_meta":1959334,"kb_avail":90254088,"statfs":{"total":96636764160,"available":92420186112,"internally_reserved":0,"allocated":2208677888,"data_stored":22529177801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1533477,"internal_metadata":2006358491},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,3,2,1,5,5],"upper_bound":256},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.8,"15min":2.2869999999999999},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":11.423,"5min":11.423,"15min":15.964},"last":0.56999999999999995},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1070000000000002,"15min":2.2850000000000001},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.27,"5min":14.27,"15min":34.176000000000002},"last":0.39000000000000001}]},{"osd":1,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":4.2910000000000004,"5min":2.544,"15min":2.532},"min":{"1min":0.28199999999999997,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.600999999999999,"5min":20.600999999999999,"15min":20.600999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":2.3199999999999998,"5min":2.1659999999999999,"15min":2.3679999999999999},"min":{"1min":0.25800000000000001,"5min":0.224,"15min":0.21299999999999999},"max":{"1min":11.028,"5min":13.462999999999999,"15min":13.842000000000001},"last":0.45700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739100,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5559644,"kb_used_data":3462356,"kb_used_omap":2774,"kb_used_meta":2094505,"kb_avail":88812196,"statfs":{"total":96636764160,"available":90943688704,"internally_reserved":0,"allocated":3545452544,"data_stored":49588494090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":2144773727},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,6,1,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":3.3799999999999999,"5min":3.1179999999999999,"15min":3.0920000000000001},"min":{"1min":0.188,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":10.433999999999999,"5min":19.873000000000001,"15min":45.886000000000003},"last":0.26100000000000001},{"interface":"front","average":{"1min":3.6880000000000002,"5min":4.0599999999999996,"15min":3.524},"min":{"1min":0.22,"5min":0.22,"15min":0.20100000000000001},"max":{"1min":13.065,"5min":20.763000000000002,"15min":45.555},"last":0.33300000000000002}]},{"osd":2,"last update":"Tue Oct 3 18:32:55 2023","interfaces":[{"interface":"back","average":{"1min":2.452,"5min":1.996,"15min":1.972},"min":{"1min":0.20899999999999999,"5min":0.186,"15min":0.186},"max":{"1min":7.9080000000000004,"5min":8.1519999999999992,"15min":13.058},"last":3.4580000000000002},{"interface":"front","average":{"1min":2.6080000000000001,"5min":2.8380000000000001,"15min":2.7530000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.192},"max":{"1min":9.7430000000000003,"5min":18.352,"15min":25.77},"last":4.1079999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739066,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7640428,"kb_used_data":5615780,"kb_used_omap":4493,"kb_used_meta":2020146,"kb_avail":86731412,"statfs":{"total":96636764160,"available":88812965888,"internally_reserved":0,"allocated":5750558720,"data_stored":72083453826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":2068629944},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,4,1,11,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":3.3460000000000001,"5min":4.6900000000000004,"15min":4.0049999999999999},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.4250000000000007,"5min":53.280000000000001,"15min":53.280000000000001},"last":7.5460000000000003},{"interface":"front","average":{"1min":2.6539999999999999,"5min":3.46,"15min":3.0800000000000001},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.5909999999999993,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.62}]},{"osd":2,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":2.298,"15min":2.4980000000000002},"min":{"1min":0.23400000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":9.8879999999999999,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.804,"5min":2.3130000000000002,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":8.5500000000000007,"5min":14.926,"15min":36.332999999999998},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3539804160,"data_stored":49589705881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5746212864,"data_stored":72078480657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2203832320,"data_stored":22528030840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:33:52.833 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:52.833 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:52.833 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:52.833 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:33:52.833 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:33:56.107 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.104+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:33:56.127 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.119+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:33:56.127 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.119+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:33:56.173 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.168+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:33:56.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.184+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:33:56.394 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.393+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:33:56.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.397+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:33:56.399 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.397+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:33:56.482 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:33:56.479+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:01.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:01.463+0000 7fc515f98700 -1 reset not still connected to 0x56221d6f3a00 2023-10-03T18:34:01.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:01.463+0000 7fc515f98700 -1 reset not still connected to 0x56222196b520 2023-10-03T18:34:01.464 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:01.463+0000 7fc515f98700 -1 reset not still connected to 0x562227e215f0 2023-10-03T18:34:01.526 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:01.525+0000 7fc515f98700 -1 reset not still connected to 0x56222cf1a680 2023-10-03T18:34:03.174 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.173+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:03.179 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.177+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:03.250 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.239+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:03.304 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.303+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:03.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.327+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:03.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.327+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:03.384 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.377+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:03.387 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.380+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:03.387 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:03.380+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:04.565 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:04.565+0000 7fc515f98700 -1 reset not still connected to 0x5622072668f0 2023-10-03T18:34:04.566 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:04.565+0000 7fc515f98700 -1 reset not still connected to 0x5622231c4b60 2023-10-03T18:34:04.596 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:04.594+0000 7fc515f98700 -1 reset not still connected to 0x562228058410 2023-10-03T18:34:04.639 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:04.638+0000 7fc515f98700 -1 reset not still connected to 0x562231e17e10 2023-10-03T18:34:05.686 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:05.679+0000 7fc515f98700 -1 reset not still connected to 0x562203b16d00 2023-10-03T18:34:05.916 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:05.915+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:05.922 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:05.919+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:06.262 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.260+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:06.463 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.462+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:06.463 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.462+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:06.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.516+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:06.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.516+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:06.520 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.516+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:06.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.550+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:06.562 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.559+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:06.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:06.777+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:07.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:07.030+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:07.031 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:07.030+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:07.217 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:07.215+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:07.259 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:07.256+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:07.367 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:07.365+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:12.834 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:34:13.478 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:34:13.485 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:34:13.501 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2316,"stamp":"2023-10-03T18:34:12.821468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24273393457,"num_objects":76106,"num_object_clones":70840,"num_object_copies":152212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76106,"num_whiteouts":0,"num_read":1582107,"num_read_kb":5689039,"num_write":1415407,"num_write_kb":5639691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80519,"ondisk_log_size":80519,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17636160,"kb_used_data":11282780,"kb_used_omap":8765,"kb_used_meta":6344578,"kb_avail":265479360,"statfs":{"total":289910292480,"available":271850864640,"internally_reserved":0,"allocated":11553566720,"data_stored":145248771831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8975886,"internal_metadata":6496848370},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,4,17,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":46,"apply_latency_ms":46,"commit_latency_ns":46000000,"apply_latency_ns":46000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5722014,"num_objects":510,"num_object_clones":512,"num_object_copies":1020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":510,"num_whiteouts":0,"num_read":3040,"num_read_kb":7223,"num_write":1525,"num_write_kb":5673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.850690"},"pg_stats":[{"pgid":"2.7","version":"480'218273","reported_seq":378446,"reported_epoch":480,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:07.410756+0000","last_change":"2023-10-03T18:34:05.367299+0000","last_active":"2023-10-03T18:34:07.410756+0000","last_peered":"2023-10-03T18:34:07.410756+0000","last_clean":"2023-10-03T18:34:07.410756+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:34:07.410756+0000","last_undegraded":"2023-10-03T18:34:07.410756+0000","last_fullsized":"2023-10-03T18:34:07.410756+0000","mapping_epoch":12,"log_start":"437'208241","ondisk_log_start":"437'208241","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10032,"log_dups_size":0,"ondisk_log_size":10032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":371,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T21:43:04.519522+0000","scrub_duration":505.38444223300002,"objects_trimmed":0,"snaptrim_duration":0.33036934099999998,"stat_sum":{"num_bytes":3023548901,"num_objects":13998,"num_object_clones":13347,"num_object_copies":27996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13998,"num_whiteouts":0,"num_read":192343,"num_read_kb":707111,"num_write":175809,"num_write_kb":700884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"204","length":"2"}]},{"pgid":"2.6","version":"480'201801","reported_seq":351153,"reported_epoch":480,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:07.466176+0000","last_change":"2023-10-03T18:34:06.733015+0000","last_active":"2023-10-03T18:34:07.466176+0000","last_peered":"2023-10-03T18:34:07.466176+0000","last_clean":"2023-10-03T18:34:07.466176+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:34:07.466176+0000","last_undegraded":"2023-10-03T18:34:07.466176+0000","last_fullsized":"2023-10-03T18:34:07.466176+0000","mapping_epoch":12,"log_start":"430'191716","ondisk_log_start":"430'191716","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":284,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.74062609999999995,"stat_sum":{"num_bytes":2793300540,"num_objects":10490,"num_object_clones":9887,"num_object_copies":20980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10490,"num_whiteouts":0,"num_read":178330,"num_read_kb":652751,"num_write":162297,"num_write_kb":647586,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"4"}]},{"pgid":"2.5","version":"480'241957","reported_seq":418053,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:07.402987+0000","last_change":"2023-10-03T18:34:06.750315+0000","last_active":"2023-10-03T18:34:07.402987+0000","last_peered":"2023-10-03T18:34:07.402987+0000","last_clean":"2023-10-03T18:34:07.402987+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:34:07.402987+0000","last_undegraded":"2023-10-03T18:34:07.402987+0000","last_fullsized":"2023-10-03T18:34:07.402987+0000","mapping_epoch":12,"log_start":"440'231917","ondisk_log_start":"440'231917","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":322,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.078163577999999997,"stat_sum":{"num_bytes":3276841614,"num_objects":13555,"num_object_clones":12845,"num_object_copies":27110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13555,"num_whiteouts":0,"num_read":221444,"num_read_kb":773840,"num_write":191558,"num_write_kb":761885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"}]},{"pgid":"2.0","version":"481'242952","reported_seq":407529,"reported_epoch":481,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:12.409079+0000","last_change":"2023-10-03T18:34:11.378736+0000","last_active":"2023-10-03T18:34:12.409079+0000","last_peered":"2023-10-03T18:34:12.409079+0000","last_clean":"2023-10-03T18:34:12.409079+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:34:12.409079+0000","last_undegraded":"2023-10-03T18:34:12.409079+0000","last_fullsized":"2023-10-03T18:34:12.409079+0000","mapping_epoch":12,"log_start":"445'232878","ondisk_log_start":"445'232878","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":155,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:42:10.295593+0000","scrub_duration":734.24538164700004,"objects_trimmed":60,"snaptrim_duration":3.3647109419999999,"stat_sum":{"num_bytes":3166458613,"num_objects":7779,"num_object_clones":7092,"num_object_copies":15558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7779,"num_whiteouts":0,"num_read":200285,"num_read_kb":737975,"num_write":184891,"num_write_kb":737166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"5"}]},{"pgid":"1.0","version":"88'99","reported_seq":648,"reported_epoch":479,"state":"active+clean","last_fresh":"2023-10-03T18:34:06.091428+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:34:06.091428+0000","last_peered":"2023-10-03T18:34:06.091428+0000","last_clean":"2023-10-03T18:34:06.091428+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:34:06.091428+0000","last_undegraded":"2023-10-03T18:34:06.091428+0000","last_fullsized":"2023-10-03T18:34:06.091428+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"480'232776","reported_seq":401387,"reported_epoch":480,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:07.332801+0000","last_change":"2023-10-03T18:34:05.367910+0000","last_active":"2023-10-03T18:34:07.332801+0000","last_peered":"2023-10-03T18:34:07.332801+0000","last_clean":"2023-10-03T18:34:07.332801+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:34:07.332801+0000","last_undegraded":"2023-10-03T18:34:07.332801+0000","last_fullsized":"2023-10-03T18:34:07.332801+0000","mapping_epoch":12,"log_start":"439'222737","ondisk_log_start":"439'222737","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10039,"log_dups_size":0,"ondisk_log_size":10039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":312,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.54996185500000005,"stat_sum":{"num_bytes":3218846436,"num_objects":12955,"num_object_clones":12258,"num_object_copies":25910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12955,"num_whiteouts":0,"num_read":203615,"num_read_kb":749574,"num_write":187123,"num_write_kb":746026,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"7"}]},{"pgid":"2.1","version":"481'231821","reported_seq":384371,"reported_epoch":481,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:12.351341+0000","last_change":"2023-10-03T18:34:11.372277+0000","last_active":"2023-10-03T18:34:12.351341+0000","last_peered":"2023-10-03T18:34:12.351341+0000","last_clean":"2023-10-03T18:34:12.351341+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:34:12.351341+0000","last_undegraded":"2023-10-03T18:34:12.351341+0000","last_fullsized":"2023-10-03T18:34:12.351341+0000","mapping_epoch":12,"log_start":"442'221772","ondisk_log_start":"442'221772","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10049,"log_dups_size":0,"ondisk_log_size":10049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":16,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:57:41.844440+0000","scrub_duration":424.08306342600002,"objects_trimmed":57,"snaptrim_duration":3.3493312390000001,"stat_sum":{"num_bytes":2934198894,"num_objects":3037,"num_object_clones":2402,"num_object_copies":6074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3037,"num_whiteouts":0,"num_read":193003,"num_read_kb":686175,"num_write":171197,"num_write_kb":681540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"4"}]},{"pgid":"2.2","version":"481'231642","reported_seq":390535,"reported_epoch":481,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:10.979537+0000","last_change":"2023-10-03T18:34:08.690876+0000","last_active":"2023-10-03T18:34:10.979537+0000","last_peered":"2023-10-03T18:34:10.979537+0000","last_clean":"2023-10-03T18:34:10.979537+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:34:10.979537+0000","last_undegraded":"2023-10-03T18:34:10.979537+0000","last_fullsized":"2023-10-03T18:34:10.979537+0000","mapping_epoch":12,"log_start":"441'221615","ondisk_log_start":"441'221615","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10027,"log_dups_size":0,"ondisk_log_size":10027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:32:33.714514+0000","scrub_duration":370.574046178,"objects_trimmed":41,"snaptrim_duration":0.70576375700000005,"stat_sum":{"num_bytes":2936152938,"num_objects":3047,"num_object_clones":2403,"num_object_copies":6094,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":204917,"num_read_kb":700546,"num_write":172266,"num_write_kb":684666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"3"}]},{"pgid":"2.4","version":"480'213195","reported_seq":368725,"reported_epoch":480,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:07.451968+0000","last_change":"2023-10-03T18:34:03.185810+0000","last_active":"2023-10-03T18:34:07.451968+0000","last_peered":"2023-10-03T18:34:07.451968+0000","last_clean":"2023-10-03T18:34:07.451968+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:34:07.451968+0000","last_undegraded":"2023-10-03T18:34:07.451968+0000","last_fullsized":"2023-10-03T18:34:07.451968+0000","mapping_epoch":12,"log_start":"432'203121","ondisk_log_start":"432'203121","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10074,"log_dups_size":0,"ondisk_log_size":10074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":293,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.29610367700000001,"stat_sum":{"num_bytes":2923455153,"num_objects":11243,"num_object_clones":10606,"num_object_copies":22486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11243,"num_whiteouts":0,"num_read":188064,"num_read_kb":680976,"num_write":170129,"num_write_kb":678044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"204","length":"4"},{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"6"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24272803089,"num_objects":76104,"num_object_clones":70840,"num_object_copies":152208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76104,"num_whiteouts":0,"num_read":1582001,"num_read_kb":5688948,"num_write":1415270,"num_write_kb":5637797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11536990208,"data_stored":145396976162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80420,"ondisk_log_size":80420,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739105,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4197776,"kb_used_data":2168968,"kb_used_omap":1497,"kb_used_meta":2027302,"kb_avail":90174064,"statfs":{"total":96636764160,"available":92338241536,"internally_reserved":0,"allocated":2221023232,"data_stored":22687994696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1533477,"internal_metadata":2075957723},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.8,"15min":2.2869999999999999},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":11.423,"5min":11.423,"15min":15.964},"last":0.29199999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1070000000000002,"15min":2.2850000000000001},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.27,"5min":14.27,"15min":34.176000000000002},"last":7.7590000000000003}]},{"osd":1,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":4.2910000000000004,"5min":2.544,"15min":2.532},"min":{"1min":0.28199999999999997,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.600999999999999,"5min":20.600999999999999,"15min":20.600999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":2.3199999999999998,"5min":2.1659999999999999,"15min":2.3679999999999999},"min":{"1min":0.25800000000000001,"5min":0.224,"15min":0.21299999999999999},"max":{"1min":11.028,"5min":13.462999999999999,"15min":13.842000000000001},"last":0.874}]}]},{"osd":0,"up_from":8,"seq":34359739104,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5659060,"kb_used_data":3477100,"kb_used_omap":2774,"kb_used_meta":2179177,"kb_avail":88712780,"statfs":{"total":96636764160,"available":90841886720,"internally_reserved":0,"allocated":3560550400,"data_stored":50242603499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":2231477855},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.6190000000000002,"15min":3.048},"min":{"1min":0.26100000000000001,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":7.4580000000000002,"5min":11.481999999999999,"15min":45.886000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":2.6629999999999998,"5min":3.7829999999999999,"15min":3.4119999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20100000000000001},"max":{"1min":13.603999999999999,"5min":20.763000000000002,"15min":45.555},"last":9.0449999999999999}]},{"osd":2,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":2.7650000000000001,"5min":2.1600000000000001,"15min":2.0550000000000002},"min":{"1min":0.245,"5min":0.186,"15min":0.186},"max":{"1min":23.027000000000001,"5min":23.027000000000001,"15min":23.027000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":2.9780000000000002,"5min":2.7829999999999999,"15min":2.8170000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.192},"max":{"1min":7.7560000000000002,"5min":18.352,"15min":25.77},"last":0.80300000000000005}]}]},{"osd":1,"up_from":8,"seq":34359739069,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7779324,"kb_used_data":5636712,"kb_used_omap":4493,"kb_used_meta":2138098,"kb_avail":86592516,"statfs":{"total":96636764160,"available":88670736384,"internally_reserved":0,"allocated":5771993088,"data_stored":72318173636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":2189412792},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,4,2,15,11,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":3.3460000000000001,"5min":4.6900000000000004,"15min":4.0049999999999999},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.4250000000000007,"5min":53.280000000000001,"15min":53.280000000000001},"last":3.6480000000000001},{"interface":"front","average":{"1min":2.6539999999999999,"5min":3.46,"15min":3.0800000000000001},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.5909999999999993,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.309}]},{"osd":2,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":2.298,"15min":2.4980000000000002},"min":{"1min":0.23400000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":9.8879999999999999,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.26500000000000001},{"interface":"front","average":{"1min":1.804,"5min":2.3130000000000002,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":8.5500000000000007,"5min":14.926,"15min":36.332999999999998},"last":9.4600000000000009}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3554639872,"data_stored":50268822730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5766766592,"data_stored":72439088401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2215583744,"data_stored":22689065031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:34:13.503 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:13.503 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:13.503 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:13.503 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:13.503 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:34:16.271 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.268+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:16.368 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.367+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:16.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.390+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:16.390 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.390+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:16.403 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.402+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:16.426 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.425+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:16.564 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:16.558+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:18.119 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.119+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:18.120 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.119+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:18.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.143+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:18.143 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.143+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:18.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.179+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:18.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.179+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:18.181 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.179+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:18.194 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.193+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:18.386 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:18.384+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:18.952 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:18.951+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:34:18.953 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:18.951+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:34:19.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:19.662+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:19.663 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:19.662+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:19.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:19.703+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:19.836 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:19.835+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:19.850 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:19.848+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:24.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:24.881+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:24.901 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:24.898+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:26.966 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:26.965+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:26.967 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:26.965+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:27.122 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.114+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:27.123 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.114+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:27.155 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:27.156 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:27.154+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:28.630 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.623+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:28.665 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.660+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:28.743 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.741+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:28.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.801+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:28.802 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.801+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:28.818 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.818+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:28.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.866+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:28.867 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:28.866+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:29.364 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.362+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:29.577 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.575+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:29.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.654+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:29.655 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.654+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:29.713 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.712+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:29.778 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.777+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:29.880 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:29.879+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:30.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:30.355+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:30.356 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:30.355+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:30.402 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:30.401+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:30.492 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:30.491+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:30.530 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:30.523+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:33.190 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.188+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:33.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.342+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:33.345 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.342+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:33.485 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.484+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:33.486 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.484+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:33.486 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.484+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:33.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:34:33.521 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.515+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:33.670 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:33.669+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2023-10-03T18:34:34.049+0000 7fa98e59c700 0 --2- 172.21.15.31:0/1378093271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9900907b0 0x7fa99012cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2023-10-03T18:34:34.446 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:34:34.457 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:34:34.478 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2329,"stamp":"2023-10-03T18:34:33.597949+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24293279537,"num_objects":77462,"num_object_clones":72196,"num_object_copies":154924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77462,"num_whiteouts":0,"num_read":1588750,"num_read_kb":5710978,"num_write":1420255,"num_write_kb":5659083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80420,"ondisk_log_size":80420,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":18066432,"kb_used_data":11321380,"kb_used_omap":8765,"kb_used_meta":6736258,"kb_avail":265049088,"statfs":{"total":289910292480,"available":271410266112,"internally_reserved":0,"allocated":11593093120,"data_stored":148508897109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8975886,"internal_metadata":6897928690},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,6,4,6,2,14,14,9],"upper_bound":512},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":10477568,"num_objects":362,"num_object_clones":362,"num_object_copies":724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":362,"num_whiteouts":0,"num_read":3764,"num_read_kb":12011,"num_write":2554,"num_write_kb":10216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.006590"},"pg_stats":[{"pgid":"2.7","version":"484'219445","reported_seq":379937,"reported_epoch":484,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:29.784032+0000","last_change":"2023-10-03T18:34:27.821459+0000","last_active":"2023-10-03T18:34:29.784032+0000","last_peered":"2023-10-03T18:34:29.784032+0000","last_clean":"2023-10-03T18:34:29.784032+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:34:29.784032+0000","last_undegraded":"2023-10-03T18:34:29.784032+0000","last_fullsized":"2023-10-03T18:34:29.784032+0000","mapping_epoch":12,"log_start":"441'209348","ondisk_log_start":"441'209348","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10097,"log_dups_size":0,"ondisk_log_size":10097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":370,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:12:31.773028+0000","scrub_duration":505.38444223300002,"objects_trimmed":20,"snaptrim_duration":0.406919278,"stat_sum":{"num_bytes":3026063845,"num_objects":14265,"num_object_clones":13614,"num_object_copies":28530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14265,"num_whiteouts":0,"num_read":193159,"num_read_kb":709951,"num_write":176421,"num_write_kb":703332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"484'202832","reported_seq":352532,"reported_epoch":484,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:29.705075+0000","last_change":"2023-10-03T18:34:27.819213+0000","last_active":"2023-10-03T18:34:29.705075+0000","last_peered":"2023-10-03T18:34:29.705075+0000","last_clean":"2023-10-03T18:34:29.705075+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:34:29.705075+0000","last_undegraded":"2023-10-03T18:34:29.705075+0000","last_fullsized":"2023-10-03T18:34:29.705075+0000","mapping_epoch":12,"log_start":"439'192820","ondisk_log_start":"439'192820","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10012,"log_dups_size":0,"ondisk_log_size":10012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":282,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":4.9422156340000001,"stat_sum":{"num_bytes":2795536956,"num_objects":10713,"num_object_clones":10110,"num_object_copies":21426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10713,"num_whiteouts":0,"num_read":179101,"num_read_kb":655368,"num_write":162843,"num_write_kb":649770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"}]},{"pgid":"2.5","version":"484'243236","reported_seq":419715,"reported_epoch":484,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:29.519409+0000","last_change":"2023-10-03T18:34:25.840912+0000","last_active":"2023-10-03T18:34:29.519409+0000","last_peered":"2023-10-03T18:34:29.519409+0000","last_clean":"2023-10-03T18:34:29.519409+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:34:29.519409+0000","last_undegraded":"2023-10-03T18:34:29.519409+0000","last_fullsized":"2023-10-03T18:34:29.519409+0000","mapping_epoch":12,"log_start":"443'233220","ondisk_log_start":"443'233220","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10016,"log_dups_size":0,"ondisk_log_size":10016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":320,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":2.9295898970000001,"stat_sum":{"num_bytes":3279635086,"num_objects":13847,"num_object_clones":13137,"num_object_copies":27694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13847,"num_whiteouts":0,"num_read":222439,"num_read_kb":776976,"num_write":192240,"num_write_kb":764613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"}]},{"pgid":"2.0","version":"484'244408","reported_seq":409279,"reported_epoch":484,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:32.575199+0000","last_change":"2023-10-03T18:34:31.318728+0000","last_active":"2023-10-03T18:34:32.575199+0000","last_peered":"2023-10-03T18:34:32.575199+0000","last_clean":"2023-10-03T18:34:32.575199+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:34:32.575199+0000","last_undegraded":"2023-10-03T18:34:32.575199+0000","last_fullsized":"2023-10-03T18:34:32.575199+0000","mapping_epoch":12,"log_start":"454'234387","ondisk_log_start":"454'234387","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10021,"log_dups_size":0,"ondisk_log_size":10021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":147,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:42:10.295593+0000","scrub_duration":734.24538164700004,"objects_trimmed":71,"snaptrim_duration":3.4754119669999999,"stat_sum":{"num_bytes":3168985845,"num_objects":7805,"num_object_clones":7118,"num_object_copies":15610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7805,"num_whiteouts":0,"num_read":201071,"num_read_kb":740691,"num_write":185508,"num_write_kb":739634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"}]},{"pgid":"1.0","version":"88'99","reported_seq":652,"reported_epoch":483,"state":"active+clean","last_fresh":"2023-10-03T18:34:22.925173+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:34:22.925173+0000","last_peered":"2023-10-03T18:34:22.925173+0000","last_clean":"2023-10-03T18:34:22.925173+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:34:22.925173+0000","last_undegraded":"2023-10-03T18:34:22.925173+0000","last_fullsized":"2023-10-03T18:34:22.925173+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"484'234056","reported_seq":402983,"reported_epoch":484,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:29.629616+0000","last_change":"2023-10-03T18:34:28.258590+0000","last_active":"2023-10-03T18:34:29.629616+0000","last_peered":"2023-10-03T18:34:29.629616+0000","last_clean":"2023-10-03T18:34:29.629616+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:34:29.629616+0000","last_undegraded":"2023-10-03T18:34:29.629616+0000","last_fullsized":"2023-10-03T18:34:29.629616+0000","mapping_epoch":12,"log_start":"442'224042","ondisk_log_start":"442'224042","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10014,"log_dups_size":0,"ondisk_log_size":10014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":310,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":14,"snaptrim_duration":2.6479304890000002,"stat_sum":{"num_bytes":3221697252,"num_objects":13274,"num_object_clones":12577,"num_object_copies":26548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13274,"num_whiteouts":0,"num_read":204458,"num_read_kb":752514,"num_write":187818,"num_write_kb":748806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"}]},{"pgid":"2.1","version":"484'233269","reported_seq":386078,"reported_epoch":484,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:32.579274+0000","last_change":"2023-10-03T18:34:30.455257+0000","last_active":"2023-10-03T18:34:32.579274+0000","last_peered":"2023-10-03T18:34:32.579274+0000","last_clean":"2023-10-03T18:34:32.579274+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:34:32.579274+0000","last_undegraded":"2023-10-03T18:34:32.579274+0000","last_fullsized":"2023-10-03T18:34:32.579274+0000","mapping_epoch":12,"log_start":"445'223184","ondisk_log_start":"445'223184","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10085,"log_dups_size":0,"ondisk_log_size":10085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:57:41.844440+0000","scrub_duration":424.08306342600002,"objects_trimmed":77,"snaptrim_duration":2.6112648919999999,"stat_sum":{"num_bytes":2936521326,"num_objects":3001,"num_object_clones":2366,"num_object_copies":6002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3001,"num_whiteouts":0,"num_read":193801,"num_read_kb":688733,"num_write":171760,"num_write_kb":683792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"a"},{"start":"4e1","length":"1"},{"start":"4e3","length":"2"}]},{"pgid":"2.2","version":"484'233060","reported_seq":392232,"reported_epoch":484,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:31.066085+0000","last_change":"2023-10-03T18:34:30.130868+0000","last_active":"2023-10-03T18:34:31.066085+0000","last_peered":"2023-10-03T18:34:31.066085+0000","last_clean":"2023-10-03T18:34:31.066085+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:34:31.066085+0000","last_undegraded":"2023-10-03T18:34:31.066085+0000","last_fullsized":"2023-10-03T18:34:31.066085+0000","mapping_epoch":12,"log_start":"445'223025","ondisk_log_start":"445'223025","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10035,"log_dups_size":0,"ondisk_log_size":10035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:32:33.714514+0000","scrub_duration":370.574046178,"objects_trimmed":48,"snaptrim_duration":2.2682480190000001,"stat_sum":{"num_bytes":2938483562,"num_objects":3059,"num_object_clones":2415,"num_object_copies":6118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3059,"num_whiteouts":0,"num_read":205771,"num_read_kb":703049,"num_write":172838,"num_write_kb":686954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"a"},{"start":"4e1","length":"1"},{"start":"4e3","length":"2"}]},{"pgid":"2.4","version":"484'214269","reported_seq":370124,"reported_epoch":484,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:29.635712+0000","last_change":"2023-10-03T18:34:28.258588+0000","last_active":"2023-10-03T18:34:29.635712+0000","last_peered":"2023-10-03T18:34:29.635712+0000","last_clean":"2023-10-03T18:34:29.635712+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:34:29.635712+0000","last_undegraded":"2023-10-03T18:34:29.635712+0000","last_fullsized":"2023-10-03T18:34:29.635712+0000","mapping_epoch":12,"log_start":"441'204228","ondisk_log_start":"441'204228","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10041,"log_dups_size":0,"ondisk_log_size":10041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":291,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":5,"snaptrim_duration":0.33380084900000001,"stat_sum":{"num_bytes":2925765297,"num_objects":11496,"num_object_clones":10859,"num_object_copies":22992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11496,"num_whiteouts":0,"num_read":188844,"num_read_kb":683605,"num_write":170690,"num_write_kb":680288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"212","length":"a"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"9"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24292689169,"num_objects":77460,"num_object_clones":72196,"num_object_copies":154920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77460,"num_whiteouts":0,"num_read":1588644,"num_read_kb":5710887,"num_write":1420118,"num_write_kb":5657189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11576078336,"data_stored":148540738082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80321,"ondisk_log_size":80321,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739109,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4311184,"kb_used_data":2176136,"kb_used_omap":1497,"kb_used_meta":2133542,"kb_avail":90060656,"statfs":{"total":96636764160,"available":92222111744,"internally_reserved":0,"allocated":2228363264,"data_stored":22971644056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1533477,"internal_metadata":2184747483},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,2,0,1,3,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":1.8,"15min":2.2869999999999999},"min":{"1min":0.21299999999999999,"5min":0.17199999999999999,"15min":0.16800000000000001},"max":{"1min":11.423,"5min":11.423,"15min":15.964},"last":4.1989999999999998},{"interface":"front","average":{"1min":2.6579999999999999,"5min":2.1070000000000002,"15min":2.2850000000000001},"min":{"1min":0.20999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":14.27,"5min":14.27,"15min":34.176000000000002},"last":5.0819999999999999}]},{"osd":1,"last update":"Tue Oct 3 18:33:33 2023","interfaces":[{"interface":"back","average":{"1min":4.2910000000000004,"5min":2.544,"15min":2.532},"min":{"1min":0.28199999999999997,"5min":0.24199999999999999,"15min":0.221},"max":{"1min":20.600999999999999,"5min":20.600999999999999,"15min":20.600999999999999},"last":4.0540000000000003},{"interface":"front","average":{"1min":2.3199999999999998,"5min":2.1659999999999999,"15min":2.3679999999999999},"min":{"1min":0.25800000000000001,"5min":0.224,"15min":0.21299999999999999},"max":{"1min":11.028,"5min":13.462999999999999,"15min":13.842000000000001},"last":4.0350000000000001}]}]},{"osd":0,"up_from":8,"seq":34359739108,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5775100,"kb_used_data":3488112,"kb_used_omap":2774,"kb_used_meta":2284201,"kb_avail":88596740,"statfs":{"total":96636764160,"available":90723061760,"internally_reserved":0,"allocated":3571826688,"data_stored":51297443643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":2339022431},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1,0,1,1,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.6190000000000002,"15min":3.048},"min":{"1min":0.26100000000000001,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":7.4580000000000002,"5min":11.481999999999999,"15min":45.886000000000003},"last":1.375},{"interface":"front","average":{"1min":2.6629999999999998,"5min":3.7829999999999999,"15min":3.4119999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20100000000000001},"max":{"1min":13.603999999999999,"5min":20.763000000000002,"15min":45.555},"last":4.6040000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":2.7650000000000001,"5min":2.1600000000000001,"15min":2.0550000000000002},"min":{"1min":0.245,"5min":0.186,"15min":0.186},"max":{"1min":23.027000000000001,"5min":23.027000000000001,"15min":23.027000000000001},"last":0.27500000000000002},{"interface":"front","average":{"1min":2.9780000000000002,"5min":2.7829999999999999,"15min":2.8170000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.192},"max":{"1min":7.7560000000000002,"5min":18.352,"15min":25.77},"last":1.411}]}]},{"osd":1,"up_from":8,"seq":34359739073,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":7980148,"kb_used_data":5657132,"kb_used_omap":4493,"kb_used_meta":2318514,"kb_avail":86391692,"statfs":{"total":96636764160,"available":88465092608,"internally_reserved":0,"allocated":5792903168,"data_stored":74239809410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":2374158776},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,6,2,3,2,12,10,4],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":3.3460000000000001,"5min":4.6900000000000004,"15min":4.0049999999999999},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.4250000000000007,"5min":53.280000000000001,"15min":53.280000000000001},"last":0.374},{"interface":"front","average":{"1min":2.6539999999999999,"5min":3.46,"15min":3.0800000000000001},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.5909999999999993,"5min":52.982999999999997,"15min":52.982999999999997},"last":0.44600000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":2.298,"15min":2.4980000000000002},"min":{"1min":0.23400000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":9.8879999999999999,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.804,"5min":2.3130000000000002,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":8.5500000000000007,"5min":14.926,"15min":36.332999999999998},"last":0.32400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3566047232,"data_stored":51312827594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5787066368,"data_stored":74254349585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2222964736,"data_stored":22973560903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:34:34.480 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:34.480 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:34.480 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:34.480 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:34.481 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-03T18:34:34.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.786+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:34.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.792+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:34.792 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.792+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:34.793 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.793+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:34.873 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.872+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:34.873 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.872+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:34.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.884+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:34.886 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.884+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:34.989 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:34.988+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:39.329 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.323+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:39.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.393+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:39.401 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.393+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:39.457 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.457+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:39.552 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.552+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:39.587 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:39.582+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:40.182 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.178+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:40.233 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.232+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:40.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.232+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:40.234 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.232+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:40.260 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.254+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:40.273 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.271+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:40.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.271+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:40.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.271+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:40.274 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.271+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:40.434 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:40.427+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:41.333 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:34:41.332+0000 7f0623ebd700 -1 reset not still connected to 0x55870c29dba0 2023-10-03T18:34:41.333 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:34:41.332+0000 7f0623ebd700 -1 reset not still connected to 0x55870c3084e0 2023-10-03T18:34:41.334 INFO:tasks.ceph.osd.0.smithi031.stderr:2023-10-03T18:34:41.332+0000 7f0623ebd700 -1 reset not still connected to 0x55870de33040 2023-10-03T18:34:42.220 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.211+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:42.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.233+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:42.245 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.233+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:42.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.233+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:42.246 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.234+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:42.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.242+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:42.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.242+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:42.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.242+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:42.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.242+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:42.247 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.242+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:42.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.978+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:42.981 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:42.978+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:43.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.000+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:43.000 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.000+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:43.014 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.012+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:43.919 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.916+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:43.920 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.916+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:43.956 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.953+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:43.971 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:43.968+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:44.937 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:44.930+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:45.002 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:44.998+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:45.003 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:44.998+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:45.080 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:45.071+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:45.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:45.071+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:45.081 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:45.071+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:45.154 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:45.151+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:45.180 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:45.177+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:48.750 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.748+0000 7fc515f98700 -1 reset not still connected to 0x562203b8c680 2023-10-03T18:34:48.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.785+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:48.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.785+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:48.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.785+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:48.795 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.786+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:48.796 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.786+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:48.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.847+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:48.854 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.847+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:48.885 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:48.881+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:49.149 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.144+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:49.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.147+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:49.150 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.147+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:49.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.161+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:49.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.209+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:49.210 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:49.209+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:49.307 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:49.303+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:34:49.319 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:49.317+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:34:49.328 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:49.327+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:34:50.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:50.166+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:50.171 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:50.166+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:50.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:50.239+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:50.241 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:50.239+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:52.606 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.597+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:52.628 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.620+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:52.667 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.659+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:52.668 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.660+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:52.681 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.672+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:52.734 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.728+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:52.736 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.733+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:52.769 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:52.755+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:52.815 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:52.813+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:34:52.832 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:52.830+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:34:52.846 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:52.842+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:34:53.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.697+0000 7fc515f98700 -1 reset not still connected to 0x562203c3da00 2023-10-03T18:34:53.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.697+0000 7fc515f98700 -1 reset not still connected to 0x562203c3dc70 2023-10-03T18:34:53.699 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.697+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:53.702 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.698+0000 7fc515f98700 -1 reset not still connected to 0x5622040912b0 2023-10-03T18:34:53.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.721+0000 7fc515f98700 -1 reset not still connected to 0x5622047b9ba0 2023-10-03T18:34:53.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.721+0000 7fc515f98700 -1 reset not still connected to 0x562204a2d1e0 2023-10-03T18:34:53.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.721+0000 7fc515f98700 -1 reset not still connected to 0x562204af4f70 2023-10-03T18:34:53.725 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.721+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:53.730 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:53.728+0000 7fc515f98700 -1 reset not still connected to 0x5622059dcd00 2023-10-03T18:34:54.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-03T18:34:55.421 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T18:34:55.445 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T18:34:55.456 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":2342,"stamp":"2023-10-03T18:34:53.605061+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":24319674161,"num_objects":77315,"num_object_clones":72049,"num_object_copies":154630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77315,"num_whiteouts":0,"num_read":1595686,"num_read_kb":5738606,"num_write":1426691,"num_write_kb":5684827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80498,"ondisk_log_size":80498,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":17041172,"kb_used_data":11371388,"kb_used_omap":8768,"kb_used_meta":5660991,"kb_avail":266074348,"statfs":{"total":289910292480,"available":272460132352,"internally_reserved":0,"allocated":11644301312,"data_stored":148496144821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8978500,"internal_metadata":5796855740},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,6,12,9,18,11,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14602240,"num_objects":-74,"num_object_clones":-74,"num_object_copies":-148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-74,"num_whiteouts":0,"num_read":3776,"num_read_kb":15032,"num_write":3561,"num_write_kb":14244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002845"},"pg_stats":[{"pgid":"2.7","version":"486'220597","reported_seq":381853,"reported_epoch":486,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:51.528234+0000","last_change":"2023-10-03T18:34:50.944760+0000","last_active":"2023-10-03T18:34:51.528234+0000","last_peered":"2023-10-03T18:34:51.528234+0000","last_clean":"2023-10-03T18:34:51.528234+0000","last_became_active":"2023-10-03T17:33:02.223695+0000","last_became_peered":"2023-10-03T17:33:02.223695+0000","last_unstale":"2023-10-03T18:34:51.528234+0000","last_undegraded":"2023-10-03T18:34:51.528234+0000","last_fullsized":"2023-10-03T18:34:51.528234+0000","mapping_epoch":12,"log_start":"444'210549","ondisk_log_start":"444'210549","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'136132","last_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_deep_scrub":"135'136132","last_deep_scrub_stamp":"2023-10-03T18:10:02.702304+0000","last_clean_scrub_stamp":"2023-10-03T18:10:02.702304+0000","objects_scrubbed":3509,"log_size":10048,"log_dups_size":0,"ondisk_log_size":10048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":366,"last_scrub_duration":506,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T01:12:31.773028+0000","scrub_duration":505.38444223300002,"objects_trimmed":54,"snaptrim_duration":8.0578690159999997,"stat_sum":{"num_bytes":3029279205,"num_objects":14301,"num_object_clones":13650,"num_object_copies":28602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14301,"num_whiteouts":0,"num_read":194082,"num_read_kb":713635,"num_write":177204,"num_write_kb":706464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2101,"num_omap_keys":25,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"214","length":"2"}]},{"pgid":"2.6","version":"486'203964","reported_seq":354298,"reported_epoch":486,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:51.531867+0000","last_change":"2023-10-03T18:34:48.787029+0000","last_active":"2023-10-03T18:34:51.531867+0000","last_peered":"2023-10-03T18:34:51.531867+0000","last_clean":"2023-10-03T18:34:51.531867+0000","last_became_active":"2023-10-03T17:33:02.223819+0000","last_became_peered":"2023-10-03T17:33:02.223819+0000","last_unstale":"2023-10-03T18:34:51.531867+0000","last_undegraded":"2023-10-03T18:34:51.531867+0000","last_fullsized":"2023-10-03T18:34:51.531867+0000","mapping_epoch":12,"log_start":"441'193922","ondisk_log_start":"441'193922","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10042,"log_dups_size":0,"ondisk_log_size":10042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":278,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":43,"snaptrim_duration":5.8984692909999996,"stat_sum":{"num_bytes":2798596668,"num_objects":10735,"num_object_clones":10132,"num_object_copies":21470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10735,"num_whiteouts":0,"num_read":179909,"num_read_kb":658572,"num_write":163590,"num_write_kb":652758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"2"}]},{"pgid":"2.5","version":"486'244586","reported_seq":421869,"reported_epoch":486,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:51.541018+0000","last_change":"2023-10-03T18:34:50.946481+0000","last_active":"2023-10-03T18:34:51.541018+0000","last_peered":"2023-10-03T18:34:51.541018+0000","last_clean":"2023-10-03T18:34:51.541018+0000","last_became_active":"2023-10-03T17:33:02.226336+0000","last_became_peered":"2023-10-03T17:33:02.226336+0000","last_unstale":"2023-10-03T18:34:51.541018+0000","last_undegraded":"2023-10-03T18:34:51.541018+0000","last_fullsized":"2023-10-03T18:34:51.541018+0000","mapping_epoch":12,"log_start":"447'234523","ondisk_log_start":"447'234523","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10063,"log_dups_size":0,"ondisk_log_size":10063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":317,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":44,"snaptrim_duration":4.6176606079999996,"stat_sum":{"num_bytes":3283366542,"num_objects":13872,"num_object_clones":13162,"num_object_copies":27744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13872,"num_whiteouts":0,"num_read":223423,"num_read_kb":780908,"num_write":193151,"num_write_kb":768257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"3"}]},{"pgid":"2.0","version":"486'246023","reported_seq":411466,"reported_epoch":486,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:52.772970+0000","last_change":"2023-10-03T18:34:51.176286+0000","last_active":"2023-10-03T18:34:52.772970+0000","last_peered":"2023-10-03T18:34:52.772970+0000","last_clean":"2023-10-03T18:34:52.772970+0000","last_became_active":"2023-10-03T17:33:02.226324+0000","last_became_peered":"2023-10-03T17:33:02.226324+0000","last_unstale":"2023-10-03T18:34:52.772970+0000","last_undegraded":"2023-10-03T18:34:52.772970+0000","last_fullsized":"2023-10-03T18:34:52.772970+0000","mapping_epoch":12,"log_start":"457'235998","ondisk_log_start":"457'235998","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'177079","last_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_deep_scrub":"255'177079","last_deep_scrub_stamp":"2023-10-03T18:19:58.659313+0000","last_clean_scrub_stamp":"2023-10-03T18:19:58.659313+0000","objects_scrubbed":6235,"log_size":10025,"log_dups_size":0,"ondisk_log_size":10025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":138,"last_scrub_duration":735,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T00:42:10.295593+0000","scrub_duration":734.24538164700004,"objects_trimmed":171,"snaptrim_duration":8.3180002660000003,"stat_sum":{"num_bytes":3172496117,"num_objects":7627,"num_object_clones":6940,"num_object_copies":15254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7627,"num_whiteouts":0,"num_read":201880,"num_read_kb":743919,"num_write":186362,"num_write_kb":743050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2318,"num_omap_keys":26,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"2"}]},{"pgid":"1.0","version":"88'99","reported_seq":654,"reported_epoch":485,"state":"active+clean","last_fresh":"2023-10-03T18:34:37.923407+0000","last_change":"2023-10-03T18:01:34.463539+0000","last_active":"2023-10-03T18:34:37.923407+0000","last_peered":"2023-10-03T18:34:37.923407+0000","last_clean":"2023-10-03T18:34:37.923407+0000","last_became_active":"2023-10-03T17:32:57.191774+0000","last_became_peered":"2023-10-03T17:32:57.191774+0000","last_unstale":"2023-10-03T18:34:37.923407+0000","last_undegraded":"2023-10-03T18:34:37.923407+0000","last_fullsized":"2023-10-03T18:34:37.923407+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'99","last_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_deep_scrub":"88'99","last_deep_scrub_stamp":"2023-10-03T18:01:34.463345+0000","last_clean_scrub_stamp":"2023-10-03T18:01:34.463345+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-05T02:05:44.389476+0000","scrub_duration":0.090934100000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"486'235287","reported_seq":404982,"reported_epoch":486,"state":"active+clean+snaptrim","last_fresh":"2023-10-03T18:34:51.521916+0000","last_change":"2023-10-03T18:34:49.504784+0000","last_active":"2023-10-03T18:34:51.521916+0000","last_peered":"2023-10-03T18:34:51.521916+0000","last_clean":"2023-10-03T18:34:51.521916+0000","last_became_active":"2023-10-03T17:33:02.223686+0000","last_became_peered":"2023-10-03T17:33:02.223686+0000","last_unstale":"2023-10-03T18:34:51.521916+0000","last_undegraded":"2023-10-03T18:34:51.521916+0000","last_fullsized":"2023-10-03T18:34:51.521916+0000","mapping_epoch":12,"log_start":"445'225247","ondisk_log_start":"445'225247","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10040,"log_dups_size":0,"ondisk_log_size":10040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":307,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":54,"snaptrim_duration":4.5018608709999999,"stat_sum":{"num_bytes":3225158372,"num_objects":13285,"num_object_clones":12588,"num_object_copies":26570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13285,"num_whiteouts":0,"num_read":205389,"num_read_kb":756230,"num_write":188663,"num_write_kb":752186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"3"}]},{"pgid":"2.1","version":"486'234503","reported_seq":387992,"reported_epoch":486,"state":"active+clean","last_fresh":"2023-10-03T18:34:52.754741+0000","last_change":"2023-10-03T18:34:42.535127+0000","last_active":"2023-10-03T18:34:52.754741+0000","last_peered":"2023-10-03T18:34:52.754741+0000","last_clean":"2023-10-03T18:34:52.754741+0000","last_became_active":"2023-10-03T17:33:02.226414+0000","last_became_peered":"2023-10-03T17:33:02.226414+0000","last_unstale":"2023-10-03T18:34:52.754741+0000","last_undegraded":"2023-10-03T18:34:52.754741+0000","last_fullsized":"2023-10-03T18:34:52.754741+0000","mapping_epoch":12,"log_start":"453'224490","ondisk_log_start":"453'224490","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"126'127392","last_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_deep_scrub":"126'127392","last_deep_scrub_stamp":"2023-10-03T18:08:42.710925+0000","last_clean_scrub_stamp":"2023-10-03T18:08:42.710925+0000","objects_scrubbed":3213,"log_size":10013,"log_dups_size":0,"ondisk_log_size":10013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":425,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T19:57:41.844440+0000","scrub_duration":424.08306342600002,"objects_trimmed":80,"snaptrim_duration":4.6881389520000001,"stat_sum":{"num_bytes":2939703918,"num_objects":2978,"num_object_clones":2343,"num_object_copies":5956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2978,"num_whiteouts":0,"num_read":194636,"num_read_kb":692049,"num_write":172534,"num_write_kb":686888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":3476,"num_omap_keys":39,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"a"},{"start":"4e1","length":"1"},{"start":"4e3","length":"6"},{"start":"4ea","length":"2"},{"start":"4ee","length":"1"}]},{"pgid":"2.2","version":"486'234300","reported_seq":394106,"reported_epoch":486,"state":"active+clean","last_fresh":"2023-10-03T18:34:51.234988+0000","last_change":"2023-10-03T18:34:43.185145+0000","last_active":"2023-10-03T18:34:51.234988+0000","last_peered":"2023-10-03T18:34:51.234988+0000","last_clean":"2023-10-03T18:34:51.234988+0000","last_became_active":"2023-10-03T17:33:02.226454+0000","last_became_peered":"2023-10-03T17:33:02.226454+0000","last_unstale":"2023-10-03T18:34:51.234988+0000","last_undegraded":"2023-10-03T18:34:51.234988+0000","last_fullsized":"2023-10-03T18:34:51.234988+0000","mapping_epoch":12,"log_start":"453'224230","ondisk_log_start":"453'224230","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'124621","last_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_deep_scrub":"119'124621","last_deep_scrub_stamp":"2023-10-03T18:07:43.395764+0000","last_clean_scrub_stamp":"2023-10-03T18:07:43.395764+0000","objects_scrubbed":2810,"log_size":10070,"log_dups_size":0,"ondisk_log_size":10070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":371,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T20:32:33.714514+0000","scrub_duration":370.574046178,"objects_trimmed":2,"snaptrim_duration":0.27793193900000002,"stat_sum":{"num_bytes":2941354858,"num_objects":2973,"num_object_clones":2329,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2973,"num_whiteouts":0,"num_read":206579,"num_read_kb":706269,"num_write":173539,"num_write_kb":689758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5128,"num_omap_keys":68,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"7"},{"start":"2bb","length":"1"},{"start":"2bf","length":"1"},{"start":"2c6","length":"a"},{"start":"2d7","length":"1"},{"start":"2d9","length":"4"},{"start":"2df","length":"3"},{"start":"2e5","length":"2"},{"start":"2ee","length":"8"},{"start":"2f7","length":"2"},{"start":"2fd","length":"1"},{"start":"301","length":"1"},{"start":"304","length":"7"},{"start":"312","length":"1"},{"start":"316","length":"7"},{"start":"31f","length":"1"},{"start":"321","length":"2"},{"start":"324","length":"1"},{"start":"327","length":"1"},{"start":"329","length":"1"},{"start":"32c","length":"1"},{"start":"32e","length":"2"},{"start":"333","length":"1"},{"start":"335","length":"2"},{"start":"338","length":"1"},{"start":"33e","length":"a"},{"start":"350","length":"1"},{"start":"352","length":"3"},{"start":"356","length":"4"},{"start":"35b","length":"1"},{"start":"360","length":"1"},{"start":"366","length":"9"},{"start":"370","length":"1"},{"start":"378","length":"1"},{"start":"37a","length":"6"},{"start":"383","length":"1"},{"start":"389","length":"2"},{"start":"38e","length":"a"},{"start":"3a1","length":"7"},{"start":"3a9","length":"3"},{"start":"3b6","length":"8"},{"start":"3bf","length":"1"},{"start":"3c2","length":"1"},{"start":"3c6","length":"2"},{"start":"3c9","length":"1"},{"start":"3cb","length":"3"},{"start":"3d1","length":"1"},{"start":"3d6","length":"2"},{"start":"3da","length":"1"},{"start":"3de","length":"8"},{"start":"3e7","length":"1"},{"start":"3e9","length":"1"},{"start":"3ef","length":"4"},{"start":"3f4","length":"2"},{"start":"3f8","length":"1"},{"start":"3fa","length":"1"},{"start":"3ff","length":"1"},{"start":"403","length":"1"},{"start":"406","length":"a"},{"start":"419","length":"4"},{"start":"41e","length":"4"},{"start":"424","length":"1"},{"start":"429","length":"1"},{"start":"42e","length":"a"},{"start":"43f","length":"2"},{"start":"443","length":"1"},{"start":"445","length":"2"},{"start":"448","length":"2"},{"start":"44b","length":"2"},{"start":"450","length":"1"},{"start":"456","length":"a"},{"start":"467","length":"4"},{"start":"46c","length":"3"},{"start":"473","length":"1"},{"start":"477","length":"2"},{"start":"47e","length":"a"},{"start":"491","length":"3"},{"start":"495","length":"3"},{"start":"499","length":"4"},{"start":"4a6","length":"a"},{"start":"4b8","length":"1"},{"start":"4bb","length":"6"},{"start":"4c4","length":"2"},{"start":"4c9","length":"1"},{"start":"4ce","length":"a"},{"start":"4e1","length":"1"},{"start":"4e3","length":"6"},{"start":"4ea","length":"2"},{"start":"4ee","length":"1"}]},{"pgid":"2.4","version":"486'215427","reported_seq":371964,"reported_epoch":486,"state":"active+clean+snaptrim_wait","last_fresh":"2023-10-03T18:34:51.497895+0000","last_change":"2023-10-03T18:34:49.496960+0000","last_active":"2023-10-03T18:34:51.497895+0000","last_peered":"2023-10-03T18:34:51.497895+0000","last_clean":"2023-10-03T18:34:51.497895+0000","last_became_active":"2023-10-03T17:33:02.223828+0000","last_became_peered":"2023-10-03T17:33:02.223828+0000","last_unstale":"2023-10-03T18:34:51.497895+0000","last_undegraded":"2023-10-03T18:34:51.497895+0000","last_fullsized":"2023-10-03T18:34:51.497895+0000","mapping_epoch":12,"log_start":"443'205329","ondisk_log_start":"443'205329","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T17:33:01.201801+0000","last_clean_scrub_stamp":"2023-10-03T17:33:01.201801+0000","objects_scrubbed":0,"log_size":10098,"log_dups_size":0,"ondisk_log_size":10098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":288,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":26,"snaptrim_duration":6.6087454729999999,"stat_sum":{"num_bytes":2929128113,"num_objects":11542,"num_object_clones":10905,"num_object_copies":23084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11542,"num_whiteouts":0,"num_read":189682,"num_read_kb":686933,"num_write":171511,"num_write_kb":683572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"214","length":"8"},{"start":"226","length":"a"},{"start":"23a","length":"a"},{"start":"24e","length":"a"},{"start":"25f","length":"2"},{"start":"264","length":"5"},{"start":"26a","length":"2"},{"start":"26f","length":"1"},{"start":"276","length":"a"},{"start":"28a","length":"7"},{"start":"292","length":"1"},{"start":"294","length":"1"},{"start":"297","length":"1"},{"start":"29e","length":"a"},{"start":"2b1","length":"1"},{"start":"2b3","length":"1"}]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":24319083793,"num_objects":77313,"num_object_clones":72049,"num_object_copies":154626,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77313,"num_whiteouts":0,"num_read":1595580,"num_read_kb":5738515,"num_write":1426554,"num_write_kb":5682933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":23,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":13023,"num_omap_keys":158,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11628355584,"data_stored":148490406434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":80767,"internal_metadata":0},"log_size":80399,"ondisk_log_size":80399,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739113,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":3049768,"kb_used_data":2185952,"kb_used_omap":1500,"kb_used_meta":862307,"kb_avail":91322072,"statfs":{"total":96636764160,"available":93513801728,"internally_reserved":0,"allocated":2238414848,"data_stored":22830155448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1536091,"internal_metadata":883003301},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,0,3,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:34:36 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.78,"15min":2.1749999999999998},"min":{"1min":0.17100000000000001,"5min":0.17100000000000001,"15min":0.16800000000000001},"max":{"1min":7.0140000000000002,"5min":11.423,"15min":15.964},"last":3.6400000000000001},{"interface":"front","average":{"1min":2.2490000000000001,"5min":2.0550000000000002,"15min":2.2360000000000002},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":7.7590000000000003,"5min":14.27,"15min":34.176000000000002},"last":5.9400000000000004}]},{"osd":1,"last update":"Tue Oct 3 18:34:36 2023","interfaces":[{"interface":"back","average":{"1min":1.881,"5min":2.5419999999999998,"15min":2.4529999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.221},"max":{"1min":6.4850000000000003,"5min":20.600999999999999,"15min":20.600999999999999},"last":5.9000000000000004},{"interface":"front","average":{"1min":1.538,"5min":2.0169999999999999,"15min":2.246},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":6.524,"5min":13.462999999999999,"15min":13.842000000000001},"last":7.5140000000000002}]}]},{"osd":0,"up_from":8,"seq":34359739112,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5839740,"kb_used_data":3502900,"kb_used_omap":2774,"kb_used_meta":2334057,"kb_avail":88532100,"statfs":{"total":96636764160,"available":90656870400,"internally_reserved":0,"allocated":3586969600,"data_stored":51410992475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2840993,"internal_metadata":2390074975},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.6190000000000002,"15min":3.048},"min":{"1min":0.26100000000000001,"5min":0.188,"15min":0.17399999999999999},"max":{"1min":7.4580000000000002,"5min":11.481999999999999,"15min":45.886000000000003},"last":0.20599999999999999},{"interface":"front","average":{"1min":2.6629999999999998,"5min":3.7829999999999999,"15min":3.4119999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20100000000000001},"max":{"1min":13.603999999999999,"5min":20.763000000000002,"15min":45.555},"last":0.28199999999999997}]},{"osd":2,"last update":"Tue Oct 3 18:33:58 2023","interfaces":[{"interface":"back","average":{"1min":2.7650000000000001,"5min":2.1600000000000001,"15min":2.0550000000000002},"min":{"1min":0.245,"5min":0.186,"15min":0.186},"max":{"1min":23.027000000000001,"5min":23.027000000000001,"15min":23.027000000000001},"last":3.3170000000000002},{"interface":"front","average":{"1min":2.9780000000000002,"5min":2.7829999999999999,"15min":2.8170000000000002},"min":{"1min":0.23699999999999999,"5min":0.221,"15min":0.192},"max":{"1min":7.7560000000000002,"5min":18.352,"15min":25.77},"last":0.40500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359739077,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":8151664,"kb_used_data":5682536,"kb_used_omap":4493,"kb_used_meta":2464626,"kb_avail":86220176,"statfs":{"total":96636764160,"available":88289460224,"internally_reserved":0,"allocated":5818916864,"data_stored":74254996898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4601416,"internal_metadata":2523777464},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1,12,6,16,10,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":3.3460000000000001,"5min":4.6900000000000004,"15min":4.0049999999999999},"min":{"1min":0.17799999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":9.4250000000000007,"5min":53.280000000000001,"15min":53.280000000000001},"last":3.1389999999999998},{"interface":"front","average":{"1min":2.6539999999999999,"5min":3.46,"15min":3.0800000000000001},"min":{"1min":0.25,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":9.5909999999999993,"5min":52.982999999999997,"15min":52.982999999999997},"last":1.1040000000000001}]},{"osd":2,"last update":"Tue Oct 3 18:33:47 2023","interfaces":[{"interface":"back","average":{"1min":2.5310000000000001,"5min":2.298,"15min":2.4980000000000002},"min":{"1min":0.23400000000000001,"5min":0.219,"15min":0.20300000000000001},"max":{"1min":9.8879999999999999,"5min":14.436999999999999,"15min":34.395000000000003},"last":0.25900000000000001},{"interface":"front","average":{"1min":1.804,"5min":2.3130000000000002,"15min":2.6080000000000001},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.186},"max":{"1min":8.5500000000000007,"5min":14.926,"15min":36.332999999999998},"last":1.242}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3581345792,"data_stored":51409894602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":32352,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5813960704,"data_stored":74254484753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":35483,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2233049088,"data_stored":22826027079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12932,"internal_metadata":0}]}} 2023-10-03T18:34:55.459 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:55.459 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:55.459 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:55.460 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-03T17:33:01.201801+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=17, tm_min=33, tm_sec=1, tm_wday=1, tm_yday=276, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=3, tm_hour=18, tm_min=1, tm_sec=30, tm_wday=1, tm_yday=276, tm_isdst=0) 2023-10-03T18:34:55.460 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1951, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1330, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-10-03T18:34:55.468 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-03T18:34:55.469 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-03T18:34:55.469 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-03T18:34:55.469 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T18:34:55.570 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:55.562+0000 7f9f21f20700 -1 reset not still connected to 0x5632d2936f70 2023-10-03T18:34:55.570 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:55.562+0000 7f9f21f20700 -1 reset not still connected to 0x5632d313c410 2023-10-03T18:34:55.570 INFO:tasks.ceph.osd.2.smithi031.stderr:2023-10-03T18:34:55.562+0000 7f9f21f20700 -1 reset not still connected to 0x5632d3244270 2023-10-03T18:34:57.160 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.150+0000 7fc515f98700 -1 reset not still connected to 0x562204090dd0 2023-10-03T18:34:57.161 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.150+0000 7fc515f98700 -1 reset not still connected to 0x562204be91e0 2023-10-03T18:34:57.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x562205986340 2023-10-03T18:34:57.168 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x5622074a4b60 2023-10-03T18:34:57.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x5622107301a0 2023-10-03T18:34:57.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x562210e615f0 2023-10-03T18:34:57.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x562225abb380 2023-10-03T18:34:57.169 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:57.160+0000 7fc515f98700 -1 reset not still connected to 0x56222be78dd0 2023-10-03T18:34:57.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-03T18:34:59.329 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x562205986340 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622059b1ba0 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622074a4b60 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x562207f4e8f0 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622107301a0 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x562210e615f0 2023-10-03T18:34:59.330 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622183768f0 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622217cb6c0 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x562225abb380 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x56222be78dd0 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622377ac000 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622377ad040 2023-10-03T18:34:59.331 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:34:59.326+0000 7fc515f98700 -1 reset not still connected to 0x5622377ad110 2023-10-03T18:35:00.753 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.750+0000 7fc515f98700 -1 reset not still connected to 0x562205986340 2023-10-03T18:35:00.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.772+0000 7fc515f98700 -1 reset not still connected to 0x5622059b1ba0 2023-10-03T18:35:00.781 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.773+0000 7fc515f98700 -1 reset not still connected to 0x5622074a4b60 2023-10-03T18:35:00.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.786+0000 7fc515f98700 -1 reset not still connected to 0x562207f4e8f0 2023-10-03T18:35:00.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.786+0000 7fc515f98700 -1 reset not still connected to 0x5622107301a0 2023-10-03T18:35:00.788 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.787+0000 7fc515f98700 -1 reset not still connected to 0x562210e615f0 2023-10-03T18:35:00.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.787+0000 7fc515f98700 -1 reset not still connected to 0x5622183768f0 2023-10-03T18:35:00.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.787+0000 7fc515f98700 -1 reset not still connected to 0x5622217cb6c0 2023-10-03T18:35:00.789 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.787+0000 7fc515f98700 -1 reset not still connected to 0x562225abb380 2023-10-03T18:35:00.803 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.801+0000 7fc515f98700 -1 reset not still connected to 0x56222be78dd0 2023-10-03T18:35:00.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.827+0000 7fc515f98700 -1 reset not still connected to 0x5622377ac000 2023-10-03T18:35:00.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.827+0000 7fc515f98700 -1 reset not still connected to 0x5622377ad040 2023-10-03T18:35:00.832 INFO:tasks.ceph.osd.1.smithi031.stderr:2023-10-03T18:35:00.827+0000 7fc515f98700 -1 reset not still connected to 0x5622377ad110 2023-10-03T18:35:01.572 INFO:tasks.ceph.osd.0:Stopped 2023-10-03T18:35:01.572 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-03T18:35:01.572 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T18:35:02.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-10-03T18:35:02.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-03T18:35:07.674 INFO:tasks.ceph.osd.1:Stopped 2023-10-03T18:35:07.675 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-03T18:35:07.675 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T18:35:07.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~5s 2023-10-03T18:35:13.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-03T18:35:13.083 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-10-03T18:35:13.777 INFO:tasks.ceph.osd.2:Stopped 2023-10-03T18:35:13.779 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-03T18:35:13.780 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-03T18:35:13.780 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T18:35:13.811 INFO:tasks.ceph.mgr.x:Stopped 2023-10-03T18:35:13.811 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-03T18:35:13.811 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-03T18:35:13.811 INFO:teuthology.orchestra.run:waiting for 300 2023-10-03T18:35:13.873 INFO:tasks.ceph.mon.a:Stopped 2023-10-03T18:35:13.873 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-03T18:35:13.874 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-10-03T18:35:14.036 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T18:35:14.037 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-03T18:35:14.488 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T18:35:14.488 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-03T18:35:14.952 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T18:35:14.952 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-03T18:35:15.148 INFO:tasks.ceph:Archiving mon data... 2023-10-03T18:35:15.149 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/data/mon.a.tgz 2023-10-03T18:35:15.150 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-10-03T18:35:15.224 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.PhoCym9n98 2023-10-03T18:35:15.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.PhoCym9n98 2023-10-03T18:35:15.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.PhoCym9n98 2023-10-03T18:35:15.523 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.PhoCym9n98 is 606KB 2023-10-03T18:35:15.621 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.PhoCym9n98 2023-10-03T18:35:15.641 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-03T18:35:15.642 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-03T18:35:15.760 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-03T18:35:15.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-03T18:35:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-03T18:35:15.804 INFO:tasks.ceph:Archiving crash dumps... 2023-10-03T18:35:15.805 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/remote/smithi031/crash 2023-10-03T18:35:15.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-03T18:35:15.870 INFO:tasks.ceph:Compressing logs... 2023-10-03T18:35:15.870 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T18:35:15.989 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109325.log 2023-10-03T18:35:15.989 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105636.log 2023-10-03T18:35:15.989 INFO:teuthology.orchestra.run.smithi031.stderr:gzip /var/log/ceph/ceph-client.admin.109325.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.109824.log 2023-10-03T18:35:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115571.log 2023-10-03T18:35:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117697.log /var/log/ceph/ceph-client.admin.109824.log: 2023-10-03T18:35:15.993 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106734.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108708.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120343.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107664.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112423.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120343.log.gz 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115664.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117835.log 2023-10-03T18:35:15.994 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108443.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110877.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106437.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115186.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111391.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110178.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111721.log 2023-10-03T18:35:15.995 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106383.log 2023-10-03T18:35:15.996 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104192.log 2023-10-03T18:35:16.002 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107878.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110902.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117066.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119087.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98762.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110732.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107467.log 2023-10-03T18:35:16.003 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98762.log.gz 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113528.log 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121155.log 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116588.log 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105656.log 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121155.log.gz 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104560.log 2023-10-03T18:35:16.004 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113942.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107619.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:gzip /var/log/ceph/ceph-client.admin.113942.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.111861.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113343.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111861.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113259.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.120583.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112117.log 2023-10-03T18:35:16.005 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117112.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120583.log.gz 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104375.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112676.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117112.log: /var/log/ceph/ceph-client.admin.104375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116486.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114660.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105781.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106400.log 2023-10-03T18:35:16.006 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119087.log.gz 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108306.log 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103778.log 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121201.log 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121256.log 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104192.log.gz 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105656.log.gz 2023-10-03T18:35:16.007 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105781.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103778.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104560.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107664.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106734.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107619.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121201.log.gz 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0%gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.107878.log.gz-- 0.0% 2023-10-03T18:35:16.008 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120823.log 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.108708.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106437.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107467.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106383.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108306.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106400.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104375.log.gz 2023-10-03T18:35:16.009 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109325.log.gz 2023-10-03T18:35:16.010 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121256.log.gz 2023-10-03T18:35:16.010 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105636.log.gz 2023-10-03T18:35:16.010 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111721.log.gz 2023-10-03T18:35:16.010 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112423.log.gz 2023-10-03T18:35:16.010 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108443.log.gz 2023-10-03T18:35:16.123 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113942.log.gz 2023-10-03T18:35:16.123 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111391.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120823.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115571.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112676.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115186.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116588.log.gz 2023-10-03T18:35:16.124 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106167.log 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110877.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115664.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116486.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.112117.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117066.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110178.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110902.log.gz 2023-10-03T18:35:16.125 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113528.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113343.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117112.log.gz -- replaced with /var/log/ceph/ceph-client.admin.109824.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110732.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113259.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111861.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117835.log.gz -- replaced with /var/log/ceph/ceph-client.admin.117697.log.gz 2023-10-03T18:35:16.126 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.114660.log.gz-- /var/log/ceph/ceph.log 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106167.log.gz 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111191.log 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108036.log 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116210.log 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111191.log.gz 2023-10-03T18:35:16.127 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108036.log.gz 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112735.log 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110502.log 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116210.log.gz 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109357.log 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112735.log.gz 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119476.log 2023-10-03T18:35:16.128 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110502.log.gz 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120286.log 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109357.log.gz 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118848.log 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119476.log.gz 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111452.log 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120286.log.gz 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105437.log 2023-10-03T18:35:16.129 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118848.log.gz 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106329.log 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111452.log.gz 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120537.log 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105437.log.gz 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119522.log 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106329.log.gz 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114043.log 2023-10-03T18:35:16.130 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120537.log.gz 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114237.log 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119522.log.gz 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112538.log 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114043.log.gz 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114237.log.gz 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108497.log 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110572.log 2023-10-03T18:35:16.131 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112538.log.gz 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105880.log 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108497.log.gz 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109978.log 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110572.log.gz 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105880.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.105880.log.gz-5 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.114902.log 2023-10-03T18:35:16.132 INFO:teuthology.orchestra.run.smithi031.stderr: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113174.log 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109978.log.gz 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108148.log 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114902.log.gz 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115048.log 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113174.log.gz 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112182.log 2023-10-03T18:35:16.133 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108148.log.gz 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116302.log 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115048.log.gz 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112182.log.gz 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119430.log 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105619.log 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116302.log.gz 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119430.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.108337.log 2023-10-03T18:35:16.134 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.119430.log.gz 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113896.log 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105619.log.gz 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106547.log 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108337.log.gz 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118415.log 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113896.log.gz 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117789.log 2023-10-03T18:35:16.135 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106547.log.gz 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118415.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.94060.log 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118415.log.gz 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109389.log 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117789.log.gz 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.94060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107549.log 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.94060.log.gz 2023-10-03T18:35:16.136 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110116.log 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109389.log.gz 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107549.log.gz 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109586.log 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110803.log 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110116.log.gz 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109586.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109586.log.gz 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-03T18:35:16.137 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106769.log 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110803.log.gz 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110440.log 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111555.log 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106769.log.gz 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110963.log 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110440.log.gz 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105914.log 2023-10-03T18:35:16.138 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111555.log.gz 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111891.log 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110963.log.gz 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105914.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.105914.log.gz-5 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.111222.log 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115765.log 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111891.log.gz 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118321.log 2023-10-03T18:35:16.139 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111222.log.gz 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116689.log 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115765.log.gz 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112651.log 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118321.log.gz 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112618.log 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116689.log.gz 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109095.log 2023-10-03T18:35:16.140 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112651.log.gz 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107096.log 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112618.log.gz 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113067.log 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109095.log.gz 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108236.log 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107096.log.gz 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110273.log 2023-10-03T18:35:16.141 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113067.log.gz 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105491.log 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108236.log.gz 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107250.log 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110273.log.gz 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120048.log 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105491.log.gz 2023-10-03T18:35:16.142 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106594.log 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107250.log.gz 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114754.log 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120048.log.gz 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106594.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104237.log 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106594.log.gz 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111753.log 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114754.log.gz 2023-10-03T18:35:16.143 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114430.log 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104237.log.gz 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111753.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111753.log.gz 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114430.log.gz 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116735.log 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112394.log 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113712.log 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116735.log.gz 2023-10-03T18:35:16.144 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108675.log 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112394.log.gz 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115866.log 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113712.log.gz 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108675.log.gz 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111815.log 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119381.log 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115866.log.gz 2023-10-03T18:35:16.145 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108107.log 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111815.log.gz 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108600.log 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119381.log.gz 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108107.log.gz 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119910.log 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108898.log 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108600.log.gz 2023-10-03T18:35:16.146 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114614.log 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119910.log.gz 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113989.log 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108898.log.gz 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114614.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114614.log.gz 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115002.log 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107923.log 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113989.log.gz 2023-10-03T18:35:16.147 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117355.log 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115002.log.gz 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105934.log 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107923.log.gz 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117355.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.117355.log.gz 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.113145.log 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109428.log 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105934.log.gz 2023-10-03T18:35:16.148 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113145.log.gz 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120776.log 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106204.log 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109428.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109428.log.gz 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108267.log 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120776.log.gz 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118995.log 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106204.log.gz 2023-10-03T18:35:16.149 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113666.log 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108267.log.gz 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108468.log 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118995.log.gz 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115232.log 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113666.log.gz 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112015.log 2023-10-03T18:35:16.150 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108468.log.gz 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105673.log 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115232.log.gz 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118949.log 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112015.log.gz 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108633.log 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105673.log.gz 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108537.log 2023-10-03T18:35:16.151 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118949.log.gz 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105764.log 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108633.log.gz 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107718.log 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108537.log.gz 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104330.log 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105764.log.gz 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119807.log 2023-10-03T18:35:16.152 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107718.log.gz 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104330.log.gz 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105835.log 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103686.log 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119807.log.gz 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113199.log 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105835.log.gz 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103686.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.103686.log.gz-5 2023-10-03T18:35:16.153 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.118748.log 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115471.log 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113199.log.gz 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121302.log 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118748.log.gz 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115471.log.gz 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106096.log 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106474.log 2023-10-03T18:35:16.154 INFO:teuthology.orchestra.run.smithi031.stderr: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.121302.log.gz 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117170.log 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106096.log.gz 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106474.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106474.log.gz 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105565.log 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118802.log 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117170.log.gz 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106059.log 2023-10-03T18:35:16.155 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105565.log.gz 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110471.log 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118802.log.gz 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112246.log 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106059.log.gz 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117881.log 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110471.log.gz 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112767.log 2023-10-03T18:35:16.156 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112246.log.gz 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112309.log 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117881.log.gz 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109005.log 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112767.log.gz 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106454.log 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112309.log.gz 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111916.log 2023-10-03T18:35:16.157 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109005.log.gz 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117263.log 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106454.log.gz 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118027.log 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111916.log.gz 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108568.log 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117263.log.gz 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114384.log 2023-10-03T18:35:16.158 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118027.log.gz 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109297.log 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105744.log 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114384.log.gz 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113574.log 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109297.log.gz 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117973.log 2023-10-03T18:35:16.159 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105744.log.gz 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105548.log 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113574.log.gz 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105727.log 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117973.log.gz 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113374.log 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105548.log.gz 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106508.log 2023-10-03T18:35:16.160 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105727.log.gz 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111064.log 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113374.log.gz 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109242.log 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106508.log.gz 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116164.log 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111064.log.gz 2023-10-03T18:35:16.161 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109242.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.109242.log.gz-5 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.114338.log 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109157.log 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116164.log.gz 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113804.log 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114338.log.gz 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109157.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.109157.log.gz-- 2023-10-03T18:35:16.162 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111984.log 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115913.log 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113804.log.gz 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106312.log 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111984.log.gz 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121017.log 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115913.log.gz 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118655.log 2023-10-03T18:35:16.163 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106312.log.gz 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111493.log 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121017.log.gz 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109728.log 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118655.log.gz 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109533.log 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111493.log.gz 2023-10-03T18:35:16.164 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98876.log 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109728.log.gz 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107197.log 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109533.log.gz 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110617.log 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98876.log.gz 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106847.log 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107197.log.gz 2023-10-03T18:35:16.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106113.log 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110617.log.gz 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106258.log 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106847.log.gz 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117495.log 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106113.log.gz 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116965.log 2023-10-03T18:35:16.166 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106258.log.gz 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116827.log 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117495.log.gz 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109617.log 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116965.log.gz 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106804.log 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116827.log.gz 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104376.log 2023-10-03T18:35:16.167 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109617.log.gz 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115425.log 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106804.log.gz 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106346.log 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104376.log.gz 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107496.log 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115425.log.gz 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108833.log 2023-10-03T18:35:16.168 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106346.log.gz 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119568.log 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107496.log.gz 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116394.log 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108833.log.gz 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103526.log 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119568.log.gz 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112146.log 2023-10-03T18:35:16.169 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116394.log.gz 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115617.log 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103526.log.gz 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117401.log 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112146.log.gz 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121109.log 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115617.log.gz 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107952.log 2023-10-03T18:35:16.170 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117401.log.gz 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109908.log 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121109.log.gz 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116110.log 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107952.log.gz 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105471.log 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109908.log.gz 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118267.log 2023-10-03T18:35:16.171 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116110.log.gz 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112340.log 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105471.log.gz 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114144.log 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118267.log.gz 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118119.log 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112340.log.gz 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110932.log 2023-10-03T18:35:16.172 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114144.log.gz 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107152.log 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118119.log.gz 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109459.log 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110932.log.gz 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108866.log 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107152.log.gz 2023-10-03T18:35:16.173 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108177.log 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109459.log.gz 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107005.log 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108866.log.gz 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108177.log: 27.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.108568.log.gz-5 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.118517.log 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108177.log.gz 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105988.log 2023-10-03T18:35:16.174 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107005.log.gz 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110211.log 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118517.log.gz 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111109.log 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105988.log.gz 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98669.log 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110211.log.gz 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106905.log 2023-10-03T18:35:16.175 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111109.log.gz 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103961.log 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98669.log.gz 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106905.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106905.log.gz 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113413.log 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103961.log.gz 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113478.log 2023-10-03T18:35:16.176 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114855.log 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113413.log.gz 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105528.log 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113478.log.gz 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113100.log 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114855.log.gz 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111261.log 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105528.log.gz 2023-10-03T18:35:16.177 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106366.log 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113100.log.gz 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109878.log 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111261.log.gz 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103732.log 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106366.log.gz 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115718.log 2023-10-03T18:35:16.178 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109878.log.gz 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117927.log 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103732.log.gz 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121063.log 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115718.log.gz 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106959.log 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117927.log.gz 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105511.log 2023-10-03T18:35:16.179 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121063.log.gz 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110646.log 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106959.log.gz 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108076.log 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105511.log.gz 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108368.log 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110646.log.gz 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108800.log 2023-10-03T18:35:16.180 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108076.log.gz 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111524.log 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108368.log.gz 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:gzip /var/log/ceph/ceph-client.admin.108800.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.108414.log 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108800.log.gz 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110700.log 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111524.log.gz 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108414.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108414.log.gz 2023-10-03T18:35:16.181 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104514.log 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114706.log 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110700.log.gz 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110409.log 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104514.log.gz 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105852.log 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114706.log.gz 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109064.log 2023-10-03T18:35:16.182 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110409.log.gz 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112476.log 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105852.log.gz 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116348.log 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109064.log.gz 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120721.log 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112476.log.gz 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119956.log 2023-10-03T18:35:16.183 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116348.log.gz 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114292.log 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120721.log.gz 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117020.log 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119956.log.gz 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111420.log 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114292.log.gz 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112704.log 2023-10-03T18:35:16.184 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117020.log.gz 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119614.log 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111420.log.gz 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119179.log 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112704.log.gz 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120869.log 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119614.log.gz 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117643.log 2023-10-03T18:35:16.185 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119179.log.gz 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104099.log 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120869.log.gz 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110087.log 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117643.log.gz 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109272.log 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104099.log.gz 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106022.log 2023-10-03T18:35:16.186 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110087.log.gz 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113850.log 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109272.log.gz 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119660.log 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106022.log.gz 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109126.log 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113850.log.gz 2023-10-03T18:35:16.187 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110848.log 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119660.log.gz 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109126.log.gz 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110347.log 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115324.log 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110848.log.gz 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104468.log 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110347.log.gz 2023-10-03T18:35:16.188 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115324.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.115324.log.gz-5 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.111163.log 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117549.log 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104468.log.gz 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118563.log 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111163.log.gz 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107746.log 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117549.log.gz 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98944.log 2023-10-03T18:35:16.189 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118563.log.gz 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106005.log 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107746.log.gz 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115140.log 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98944.log.gz 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113228.log 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106005.log.gz 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110242.log 2023-10-03T18:35:16.190 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115140.log.gz 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116542.log 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113228.log.gz 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110242.log.gz 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109687.log 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116542.log.gz 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116781.log 2023-10-03T18:35:16.191 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109947.log 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109687.log.gz 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111366.log 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116781.log.gz 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110671.log 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109947.log.gz 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109656.log 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111366.log.gz 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104054.log 2023-10-03T18:35:16.192 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110671.log.gz 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106629.log 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109656.log.gz 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106221.log 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104054.log.gz 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114089.log 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106629.log.gz 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106221.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.106221.log.gz-5 2023-10-03T18:35:16.193 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.113312.log 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108968.log 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114089.log.gz 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113312.log.gz 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107521.log 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108968.log.gz 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103962.log 2023-10-03T18:35:16.194 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108939.log 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107521.log.gz 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112937.log 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103962.log.gz 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108939.log.gz 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115517.log 2023-10-03T18:35:16.195 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112937.log.gz 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109757.log 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114522.log 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115517.log.gz 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105454.log 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109757.log.gz 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105582.log 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114522.log.gz 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105454.log.gz 2023-10-03T18:35:16.196 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110764.log 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115094.log 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105582.log.gz 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111784.log 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110764.log.gz 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106275.log 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115094.log.gz 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117217.log 2023-10-03T18:35:16.197 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111784.log.gz 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106275.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.120002.log 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.106275.log.gz 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106184.log 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117217.log.gz 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112883.log 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120002.log.gz 2023-10-03T18:35:16.198 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119281.log 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106184.log.gz 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112883.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112883.log.gzgzip 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.112798.log 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119281.log.gz 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121348.log 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112798.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.112798.log.gz 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.109558.log 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.121348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107777.log 2023-10-03T18:35:16.199 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.121348.log.gz 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109558.log.gz 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107808.log 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113758.log 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107777.log.gz 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107808.log.gz 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111654.log 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105897.log 2023-10-03T18:35:16.200 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113758.log.gz 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120390.log 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111654.log.gz 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111629.log 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105897.log.gz 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106042.log 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120390.log.gz 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114476.log 2023-10-03T18:35:16.201 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111629.log.gz 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110314.log 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106042.log.gz 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106664.log 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114476.log.gz 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106238.log 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110314.log.gz 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117309.log 2023-10-03T18:35:16.202 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106664.log.gz 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116919.log 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106238.log.gz 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110372.log 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117309.log.gz 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116919.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.116919.log.gz--verbose 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:-- /var/log/ceph/ceph-client.admin.107580.log 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110142.log 2023-10-03T18:35:16.203 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110372.log.gz 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120140.log 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107580.log.gz 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110142.log.gz 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107693.log 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116062.log 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120140.log.gz 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107051.log 2023-10-03T18:35:16.204 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107693.log.gz 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107287.log 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116062.log.gz 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109197.log 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107051.log.gz 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105710.log 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107287.log.gz 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115960.log 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109197.log.gz 2023-10-03T18:35:16.205 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113027.log 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105710.log.gz 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112965.log 0.0% 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.115960.log.gz 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107342.log 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113027.log.gz 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112965.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113620.log 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112965.log.gz 2023-10-03T18:35:16.206 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98496.log 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107342.log.gz 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.113620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.113620.log.gz 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118701.log 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103623.log 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98496.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98496.log.gz 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111682.log 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118701.log.gz 2023-10-03T18:35:16.207 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103623.log.gz 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112277.log 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112569.log 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111682.log.gz 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119327.log 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112277.log.gz 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107388.log 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112569.log.gz 2023-10-03T18:35:16.208 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119715.log 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119327.log.gz 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104100.log 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107388.log.gz 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119715.log.gz 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98716.log 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116440.log 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104100.log.gz 2023-10-03T18:35:16.209 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.98716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.98716.log.gz 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116256.log 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119234.log 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116440.log.gz 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109792.log 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116256.log.gz 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118073.log 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119234.log.gz 2023-10-03T18:35:16.210 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109792.log.gz 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105798.log 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103824.log 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118073.log.gz 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105798.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105798.log.gz 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112837.log 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120445.log 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103824.log.gz 2023-10-03T18:35:16.211 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120971.log 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112837.log.gz 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105968.log 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120445.log.gz 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120971.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.120971.log.gz--verbose 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:-- /var/log/ceph/ceph-client.admin.120240.log 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119041.log 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105968.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105968.log.gz 2023-10-03T18:35:16.212 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106076.log 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120240.log.gz 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119041.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119041.log.gz 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115371.log 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117596.log 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106076.log.gz 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116873.log 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115371.log.gz 2023-10-03T18:35:16.213 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104238.log 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117596.log.gz 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116015.log 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116873.log.gz 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.104238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114956.log 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.104238.log.gz 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.116015.log.gzgzip 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.106528.log 2023-10-03T18:35:16.214 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112215.log 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114956.log.gz 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110010.log 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106528.log.gz 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111033.log 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112215.log.gz 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110994.log 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110010.log.gz 2023-10-03T18:35:16.215 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111033.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.111033.log.gz 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.111600.log 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110533.log 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110994.log.gz 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120629.log 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111600.log.gz 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.110533.log.gz 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115278.log 2023-10-03T18:35:16.216 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119133.log 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120629.log.gz 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115278.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.115278.log.gz-5 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.112448.log 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118174.log 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.119133.log.gz 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111292.log 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112448.log.gz 2023-10-03T18:35:16.217 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118174.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.118174.log.gz 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.118221.log 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118895.log 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111292.log.gz 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118221.log.gz 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105690.log 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118895.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118895.log.gz 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118368.log 2023-10-03T18:35:16.218 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105690.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106699.log 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105690.log.gz 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108005.log 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118368.log.gz 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120094.log 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106699.log.gz 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120491.log 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108005.log.gz 2023-10-03T18:35:16.219 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120094.log.gz 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106150.log 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120675.log 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120491.log.gz 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106150.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.106150.log.gz/var/log/ceph/ceph-client.admin.112046.log 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120675.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.120675.log.gz-- 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118469.log 2023-10-03T18:35:16.220 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105951.log 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112046.log.gz 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118469.log.gz 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116634.log 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105951.log.gz 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111953.log 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.116634.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.116634.log.gz/var/log/ceph/ceph-client.admin.112912.log 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.221 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105818.log 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111953.log.gz 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111337.log 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112912.log.gz 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120194.log 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105818.log.gz 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114809.log 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111337.log.gz 2023-10-03T18:35:16.222 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108769.log 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120194.log.gz 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120925.log 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114809.log.gz 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108769.log.gz 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105602.log 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.120925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115811.log 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.120925.log.gz 2023-10-03T18:35:16.223 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105602.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.105602.log.gz 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.115811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106292.log 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.115811.log.gz 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114191.log 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119761.log 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106292.log.gz 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112507.log 2023-10-03T18:35:16.224 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114191.log.gz 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119761.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.103870.log -- replaced with /var/log/ceph/ceph-client.admin.119761.log.gz 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105420.log 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112507.log.gz 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107977.log 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.103870.log.gz 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.105420.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.105420.log.gz 2023-10-03T18:35:16.225 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.110041.log 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106130.log 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107977.log.gz 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.110041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109033.log 0.0% 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.110041.log.gz 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106491.log 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106130.log.gz 2023-10-03T18:35:16.226 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109033.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.117447.log 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr: -- replaced with /var/log/ceph/ceph-client.admin.109033.log.gz 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112087.log 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106491.log.gz 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117447.log.gz 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108206.log 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109853.log 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112087.log.gz 2023-10-03T18:35:16.227 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109504.log 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108206.log.gz 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118609.log 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109853.log.gz 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.109504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107847.log 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.109504.log.gz 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.118609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103960.log 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.118609.log.gz 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.107847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119864.log 2023-10-03T18:35:16.228 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.107847.log.gz 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.103960.log: gzip -5 0.0%--verbose -- replaced with /var/log/ceph/ceph-client.admin.103960.log.gz-- 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114568.log 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.119864.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.119864.log.gz 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:--verbose -- /var/log/ceph/ceph-client.admin.112996.log 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.114568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108733.log 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.114568.log.gz 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.112996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111138.log 2023-10-03T18:35:16.229 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.112996.log.gz 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.108733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117743.log 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.108733.log.gz 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.111138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.111138.log.gz 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106420.log 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.117743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.117743.log.gz 2023-10-03T18:35:16.230 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph-client.admin.106420.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.106420.log.gz 2023-10-03T18:35:16.394 INFO:teuthology.orchestra.run.smithi031.stderr: 90.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-03T18:35:18.085 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-03T18:35:20.699 INFO:teuthology.orchestra.run.smithi031.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-03T18:35:21.075 INFO:teuthology.orchestra.run.smithi031.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-03T18:40:49.241 INFO:teuthology.orchestra.run.smithi031.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-03T18:41:25.906 INFO:teuthology.orchestra.run.smithi031.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-03T18:46:38.986 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-03T18:46:39.006 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T18:46:39.006 INFO:teuthology.orchestra.run.smithi031.stderr:real 11m23.063s 2023-10-03T18:46:39.007 INFO:teuthology.orchestra.run.smithi031.stderr:user 12m22.948s 2023-10-03T18:46:39.007 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m27.462s 2023-10-03T18:46:39.007 INFO:tasks.ceph:Archiving logs... 2023-10-03T18:46:39.010 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/remote/smithi031/log 2023-10-03T18:46:39.011 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T18:49:27.719 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1951, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1330, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-10-03T18:49:27.898 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-03T18:49:27.908 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1951, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1330, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-10-03T18:49:27.910 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-03T18:49:27.910 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-03T18:49:27.998 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-03T18:49:28.008 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-03T18:49:28.008 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T18:49:28.027 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-10-03T18:49:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T18:49:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-03T18:49:28.094 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -364us[ -339us] +/- 82ms 2023-10-03T18:49:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 56 +1605us[+1630us] +/- 70ms 2023-10-03T18:49:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 -1680us[-1655us] +/- 88ms 2023-10-03T18:49:28.095 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T18:49:28.100 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-03T18:49:28.109 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-03T18:49:28.110 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-03T18:49:28.119 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-03T18:49:28.334 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T18:49:28.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T18:49:28.369 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-03T18:49:28.382 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-03T18:49:28.393 INFO:teuthology.task.internal:Duration was 5495.970624 seconds 2023-10-03T18:49:28.393 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-03T18:49:28.415 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-03T18:49:28.415 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T18:49:28.564 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T18:49:28.619 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-03T18:49:28.619 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T18:49:28.619 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T18:49:28.697 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-03T18:49:28.697 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T18:49:28.766 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-03T18:49:28.766 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T18:49:28.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-03T18:49:28.884 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-03T18:49:28.884 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T18:49:28.931 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-03T18:49:28.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T18:49:29.011 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-10-03T18:49:29.036 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T18:49:29.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T18:49:29.094 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-03T18:49:29.106 INFO:teuthology.task.internal:Transferring archived files... 2023-10-03T18:49:29.111 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690/remote/smithi031 2023-10-03T18:49:29.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T18:49:29.200 INFO:teuthology.task.internal:Removing archive directory... 2023-10-03T18:49:29.200 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T18:49:29.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-03T18:49:29.239 INFO:teuthology.task.internal:Not uploading archives. 2023-10-03T18:49:29.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-03T18:49:29.252 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-03T18:49:29.253 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T18:49:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: 262307 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 3 18:49 /home/ubuntu/cephtest 2023-10-03T18:49:29.303 INFO:teuthology.orchestra.run.smithi031.stdout: 659413 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 17:33 /home/ubuntu/cephtest/mnt.0 2023-10-03T18:49:29.304 INFO:teuthology.orchestra.run.smithi031.stdout: 659414 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 17:34 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-03T18:49:29.304 INFO:teuthology.orchestra.run.smithi031.stdout: 659456 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 3 17:34 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-03T18:49:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T18:49:29.306 INFO:teuthology.orchestra.run.smithi031.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-03T18:49:29.306 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1951, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_a7de91bfa5778a0d559a7e9bc8e35f93456b41ad/qa/tasks/ceph.py", line 1330, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T18:49:29.319 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-03T18:49:29.338 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-03T18:49:29.356 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2023-10-03T18:49:29.356 INFO:teuthology.nuke:Checking targets against current locks 2023-10-03T18:49:29.409 DEBUG:teuthology.nuke:shortname: smithi031 2023-10-03T18:49:29.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T18:49:29.433 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rraja-2023-10-03_17:10:36-rbd:cli-wip-ajarr-63009-only-reacquire-kickstart-distro-default-smithi/7409690', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 17:11:34.330721', 'locked_by': 'scheduled_rraja@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-10-03T18:49:29.512 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-03T18:49:29.512 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T18:49:29.578 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-03T18:49:29.578 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T18:49:29.635 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-03T18:49:33.637 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T18:49:33.788 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-03T18:49:37.790 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T18:49:37.920 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-03T18:49:38.021 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-03T18:49:38.127 INFO:teuthology.run:Summary data: description: rbd:cli/{base/install clusters/{fixed-1 openstack} conf/{disable-pool-app} features/layering msgr-failures/few objectstore/bluestore-hybrid pool/none supported-random-distro$/{rhel_8} workloads/rbd_support_module_recovery} duration: 5495.97062420845 failure_reason: 'Command failed (workunit test rbd/rbd_support_module_recovery.sh) on smithi031 with status 22: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a7de91bfa5778a0d559a7e9bc8e35f93456b41ad TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd_support_module_recovery.sh''' flavor: default owner: scheduled_rraja@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=aabfa6783e8e4e5a9628900ae137d4a1 status: fail success: false 2023-10-03T18:49:38.128 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T18:49:38.216 INFO:teuthology.run:FAIL